Urban rail transit train positioning method fusing vision / inertial navigation

Abstract

The invention provides an urban rail transit train positioning method fusing vision / inertial navigation, and the method comprises the steps: calling an enhanced generation model with a multi-scale cavity convolution layer to extract a plurality of convolution feature maps of a low-illumination image, carrying out the channel attention mechanism processing, and obtaining a channel feature map, then generating the channel feature map to obtain a generated image, and performing threshold segmentation processing on the generated image to obtain an enhanced image; determining a key frame input into a preset sliding window from the enhanced image; and calculating the visual residual error of the feature points between the key frames according to the feature points of the key frames, and solving the minimum value by combining the calculated inertial navigation error so as to determine the optimal pose estimation value of the train. Therefore, the defect of low train positioning precision in a subway operation environment due to the fact that the effect of estimating camera motion between two frames of images is severely influenced is overcome.

Description

Technical Field

[0001] This invention relates to the field of visual inertial navigation technology, and more particularly to a method for merging visual and inertial navigation for urban rail transit train positioning. Background Art

[0002] With the continuous advancement of urban modernization in my country, urban rail transit has developed rapidly. Due to its strong transport capacity, safety, punctuality, comfort, and energy efficiency, urban rail transit is gradually occupying an increasingly important position in urban transportation. Currently, most subway safety control systems use CBTC (Communications Based Train Control). Train position information is crucial for the CBTC system. CBTC achieves train positioning by combining onboard relative positioning and ground absolute position correction equipment. Onboard positioning mainly relies on wheel speedometers and odometers to measure train speed and distance, while ground position correction typically uses track circuits and transponders installed at fixed locations on the ground to correct the train's position. ATP (Automatic Train Protection) relies on train position information to generate MA (Movement Authority) to ensure safe train operation. ATO (Automatic Train Operation) relies on train position information to generate speed-distance curves under ATP protection to guide train operation.

[0003] The CBTC system heavily relies on the ground control center, which is responsible for train safety. Train location information must first be sent to the ground control center before dispatchers can make unified scheduling and operational arrangements. CBTC uses moving block signaling, which means the control center can determine the location of a train within one or more block sections based on their occupancy, but cannot obtain the train's exact geographical location between two stations.

[0004] Traditional CBTC train positioning relies primarily on wheel speed meters, visual odometers, ground transponders, and track circuits. However, this approach suffers from low efficiency in vehicle-to-ground information transmission, cumulative errors in positioning results, and inconvenient management and maintenance. Furthermore, the operating environment of subway trains is characterized by uneven lighting, low contrast, and significant random noise. While visual odometers offer high positioning accuracy under normal lighting conditions, their performance deteriorates in low-light scenarios. In low-light conditions, the inability to extract reliable visual image features severely impacts the estimation of camera motion between two frames, resulting in low train positioning accuracy in subway operating environments. Summary of the Invention

[0005] This invention provides a fusion vision / inertial navigation urban rail transit train positioning method to solve the problem that in the prior art, visual odometers cannot extract reliable visual image feature information in low-light scenarios during subway operation, which seriously affects the estimation of camera motion between two frames and leads to low train positioning accuracy in subway operation environments.

[0006] This invention provides a method for merging vision and inertial navigation in urban rail transit train positioning, the method comprising: The low-light image to be enhanced is acquired by the train, and an enhancement generation model with multi-scale dilated convolutional layers is called to extract multiple convolutional feature maps of the low-light image. The convolutional feature maps extracted from each multi-scale dilated convolutional layer are processed by a channel attention mechanism to obtain channel feature maps. The enhanced generation model is then called to generate the channel feature maps to obtain the generated image. The generated image is subjected to threshold segmentation to obtain an enhanced image, and keyframes are determined from the enhanced image and input into a preset sliding window. Within the sliding window, the visual residual between feature points of key frames is calculated based on the feature points of the key frames, and the inertial navigation error is calculated based on the train's inertial navigation data. When the visual-inertial navigation error reaches its minimum value, the optimal pose estimate of the train is determined. The visual-inertial navigation error is the sum of the visual residual and the inertial navigation error.

[0007] In some embodiments, the step of invoking an enhanced generative model with multi-scale dilated convolutional layers to extract convolutional feature maps of the low-light image includes: The low-light image is processed by dilated convolution of multiple multi-scale dilated convolutional layers in the enhanced generative model to obtain the convolutional feature map output by each multi-scale dilated convolutional layer. The dilated convolution process of multi-scale dilated convolutional layers includes: The low-light image is processed by a convolutional network with a 1×1 kernel to obtain the first input feature map; The low-light image is processed by dilated convolutional networks with different linear dilation factors to obtain multiple second input feature maps. The first input feature map is stacked with each of the second input feature maps, and the resulting stacked feature map is input into a convolutional network with a 5×5 kernel for convolution processing to obtain a new feature map. The new feature map and the low-light image are input into a convolutional network with a 1×1 kernel for convolution processing to obtain a convolutional feature map.

[0008] In some embodiments, the process of applying a channel attention mechanism to the convolutional feature maps extracted from each multi-scale dilated convolutional layer to obtain channel feature maps includes: The convolutional feature maps extracted from each multi-scale dilated convolutional layer are compressed to obtain the spatial features of each corresponding channel. The spatial features are stimulated to obtain the channel attention weights for each layer of channels; The channel attention weights are used to weight the feature maps of each channel in the convolutional feature map to obtain the corresponding channel feature maps.

[0009] In some embodiments, the step of performing threshold segmentation on the generated image to obtain an enhanced image includes: A first pixel threshold and a second pixel threshold are preset for threshold segmentation, wherein the first pixel threshold is less than the second pixel threshold; For each pixel in the generated image, if the gray value of the pixel is less than the first pixel threshold, the gray value of the pixel after segmentation is set to 0. When the gray value of the pixel is greater than the first pixel threshold and less than the second pixel threshold, the ratio of the gray value of the pixel to 255 is used as the gray value of the pixel after segmentation. When the gray value of the pixel is greater than the second pixel threshold, the ratio of the gray value of the pixel to 255 is determined, and the difference between 1 and the ratio is determined as the gray value of the pixel after segmentation. The enhanced image is determined based on the segmented grayscale values ​​of the pixels.

[0010] In some embodiments, determining the keyframes input into a preset sliding window from the enhanced image includes: For each current frame in the enhanced image, if the current frame satisfies one of the following conditions, then the current frame is determined as a key frame following a historical key frame within a preset sliding window. The conditions mentioned include: The number of image frames between the current frame and the historical keyframe is greater than a preset frame number threshold. The train running distance between the current frame and the historical key frame is greater than a preset distance threshold; The current frame and the historical keyframe do not have any shared feature points. The shared feature points are the feature points that are common to both the current frame and the historical keyframe.

[0011] In some embodiments, calculating the visual residual of feature points between keyframes based on the feature points of the keyframes includes: Determine the region of interest (ROI) in the keyframe; The inner reprojection error between keyframes is calculated based on the inner feature points within the rectangular region of interest, and the outer reprojection error between keyframes is calculated based on the outer feature points outside the rectangular region of interest, wherein the inner feature points and the outer feature points are assigned different feature weights. The sum of the inner reprojection error and the outer reprojection error is used as the visual residual of feature points between keyframes.

[0012] In some embodiments, calculating the inertial navigation error based on the train's inertial navigation data includes: The inertial residuals are calculated based on the adjacent IMU measurements in the train's inertial navigation data and the pose to be optimized. Obtain the marginalized prior residuals in the sliding window, and determine the position increment of the pose to be optimized based on the Jacobian matrix of the prior residuals; The position error of the pose to be optimized is determined based on the difference between the prior residual and the position increment. The sum of the inertial residual and the position error is taken as the inertial navigation error.

[0013] In some embodiments, the minimum value of the visual-inertial navigation error is obtained by iterative calculation within the sliding window after setting the descent step size of the train's pose to be optimized. Iterative calculation of the minimum value of the visual-inertial navigation error based on the descent step size of the pose to be optimized includes: Determine the initial values ​​for the pose to be optimized; In each iteration, the increment of the initial value is determined based on the descent step size, and the minimum value of the visual-inertial navigation error is calculated based on the increment. When the incremental value drops to the incremental threshold, the iteration stops, and the minimum value is taken as the minimum value of the visual inertial navigation error. If the incremental value does not decrease to the incremental threshold, the incremental value is used as the initial value of the pose to be optimized in the next iteration. When the gain ratio of the minimum value is greater than 0, it is determined to enter the next iteration. When the gain ratio is greater than the preset gain threshold, the decreasing step size of the incremental value in the next iteration is increased.

[0014] In some embodiments, the process of determining the descent step size of the pose to be optimized includes: The trust dog leg algorithm is used to calculate the first descent step size under the steepest gradient method and the second descent step size under the Gauss-Newton method, respectively. When the second descent step size is not greater than the radius of the trust region set in the trust dog leg algorithm, the second descent step size is used as the descent step size of the pose to be optimized. When the first descent step size is not less than the trust region radius, the unit step size value of the first descent step size is determined, and the product of the unit step size value and the trust region radius is used as the descent step size of the pose to be optimized. When the radius of the trust region is not less than the first descent step size and not greater than the second descent step size, the step size difference between the first descent step size and the second descent step size is determined; The step size difference is weighted by a preset interpolation weight, and the weighted result is summed with the first descent step size to obtain the descent step size of the pose to be optimized.

[0015] The present invention also provides a vision / inertial navigation integrated urban rail transit train positioning device, the device comprising: The extraction module is used to acquire low-light images to be enhanced taken by the train, and call an enhancement generation model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps of the low-light images. The generation module is used to process the convolutional feature maps extracted from each multi-scale dilated convolutional layer using a channel attention mechanism to obtain channel feature maps, and then calls the enhanced generation model to generate the channel feature maps to obtain the generated image. An enhancement module is used to perform threshold segmentation on the generated image to obtain an enhanced image, and to determine keyframes from the enhanced image that are input into a preset sliding window. The determination module is used to calculate the visual reprojection error of feature points between key frames based on the feature points of the key frames within the sliding window, and to calculate the inertial navigation error based on the inertial navigation data of the train. The positioning module is used to determine the optimal pose estimate of the train when the visual-inertial navigation error reaches its minimum value. The visual-inertial navigation error is the sum of the visual reprojection error and the inertial navigation error.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the urban rail transit train positioning method fused with vision / inertial navigation as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the urban rail transit train positioning method fused with vision / inertial navigation as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the urban rail transit train positioning method fused with vision / inertial navigation as described above.

[0019] The present invention provides a vision / inertial navigation-based urban rail transit train localization method. By performing dilated convolution on the low-light image to be enhanced captured by the train, the receptive field of the network can be increased, and more image context information can be fused. Channel attention is calculated during the dilated convolution process, enabling adaptive adjustment of image channels and reducing the computational load of the model. Thresholding segmentation of the generated image overcomes the limitation of enhancement models that enhance low-pixel counts while weakening high-pixel counts, which is often ineffective in train localization. Finally, the enhanced image is used for vision / inertial navigation-based train localization to determine the optimal pose estimate and improve the train's localization accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the urban rail transit train positioning method that integrates vision and inertial navigation provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the enhanced generative model provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the processing of the dilated convolution module provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the channel attention processing provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the sliding window for calculating pose provided by the present invention.

[0026] Figure 6 This is a calculation diagram of the trust dog-leg method provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the urban rail transit train positioning device that integrates vision and inertial navigation provided by the present invention.

[0028] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] The urban rail transit train positioning method integrating vision / inertial navigation of the present invention can be deployed and applied to the server or terminal of the CBTC safety control system of the subway, and can also be applied to the ATP or ATO of the train. This method can provide real-time and accurate positioning of rail transit trains to guide train operation and route planning.

[0031] The following description, in conjunction with the accompanying drawings, illustrates the urban rail transit train positioning method of the present invention, which integrates visual and inertial navigation. Figure 1 This is a flowchart illustrating the urban rail transit train positioning method integrating vision and inertial navigation provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 to 105.

[0032] Step 101: Obtain the low-light image to be enhanced taken by the train, and call the enhancement generation model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps of the low-light image.

[0033] Positioning devices such as camera sensors and visual-inertial odometers are installed on rail transit trains. During train operation, the installed camera sensors capture and upload low-light images in real time, including those taken on roads, bridges, or tunnels, such as in poor weather conditions or under bridges and tunnels.

[0034] After the low-light image is uploaded, the low-light image to be enhanced, captured by the train, is obtained. Then, an enhancement generative model with multi-scale atrous convolutional layers is called to extract multiple convolutional feature maps from the low-light image. This enhancement generative model can be the EnlightenGAN model, whose backbone network is a U-Net structure. In this embodiment, all ordinary convolutional modules in the U-Net structure are replaced with multi-scale atrous convolutional modules (MACM), and an attention module is set during feature extraction to extract image features from the low-light image. During feature extraction, the enhancement generative model with multi-scale atrous convolutional layers is called to extract multiple convolutional feature maps from the low-light image.

[0035] First, if Figure 2As shown, in the U-Net structure of the EnlightenGAN model, during feature extraction, convolutional features of the input image are extracted sequentially through convolutional modules. Each convolutional module obtains a feature map of a different size, and the convolutional modules are connected through residual connections to avoid information loss.

[0036] In this embodiment of the invention, multiple convolutional feature maps of the low-light image are extracted by calling an enhancement generative model with multi-scale dilated convolutional layers. The low-light image to be enhanced is input into the U-Net structure of the EnlightenGAN model, and multiple convolutional feature maps of the low-light image are extracted through its multi-scale dilated convolutional layers.

[0037] Specifically, the low-light image is processed by dilated convolution of multiple multi-scale dilated convolutional layers in the enhanced generative model. The processing result of each multi-scale dilated convolutional layer is input into the next multi-scale dilated convolutional layer, and finally the convolutional feature map output by each multi-scale dilated convolutional layer is obtained.

[0038] like Figure 3 As shown, the dilated convolution process of the multi-scale dilated convolutional layer includes: First, a convolutional network with a 1×1 kernel is used to convolve the low-light image to obtain a first input feature map. Then, multiple dilated convolutional networks with different linear dilation factors are used to dilate the low-light image to obtain multiple second input feature maps. Next, the first input feature maps are stacked with each of the second input feature maps, and the resulting stacked feature map is input into a convolutional network with a 5×5 kernel for convolution to obtain a new feature map. Finally, the new feature map and the low-light image are input into a convolutional network with a 1×1 kernel for convolution to obtain a convolutional feature map.

[0039] Here, by setting different linear dilation factors during the dilated convolution process, the receptive field of the EnlightenGAN model for low-light images can be increased. By extracting features of different scales through different linear dilation factors and stacking them, more image context information can be fused, thereby improving the accuracy of feature extraction.

[0040] Step 102: Process the convolutional feature maps extracted from each multi-scale dilated convolutional layer using the channel attention mechanism to obtain channel feature maps, and then call the enhanced generation model to generate the channel feature maps to obtain the generated image.

[0041] like Figure 2As shown in the diagram, the feature extraction process in the multi-scale dilated convolutional layer is illustrated. In this embodiment of the invention, a channel attention module is added at the end of the multi-scale dilated convolutional layer. This channel attention mechanism module operates on each multi-scale dilated convolutional layer, processing the convolutional feature map extracted from each layer to obtain the channel feature map. The following section will further explain this process. Figure 4 Detailed explanation.

[0042] First, the convolutional feature maps extracted from each multi-scale dilated convolutional layer are compressed to obtain the spatial features of each corresponding channel. For example... Figure 4 As shown, the compression process is essentially feature compression. Here, we assume that the height, width, and channel size of the convolutional feature map U of the low-light image X are H*W*C. Through the compression operation, the convolutional feature map is compressed into a spatial feature with a height, width, and channel size of 1*1*C, denoted as H*W*C. The squeezing operation can be implemented by mapping through a fully connected neural network, as shown in the following formula (1): (1) In the above formula (1), Let U represent the feature map of the c-th channel of the convolutional feature map U. The coordinates of the c-th channel in the convolutional feature map U are... The pixels are H and W, which represent the height and width of the convolutional feature map U, respectively. This represents the mapping function used for extrusion.

[0043] Next, spatial features The activation operation is performed to obtain the channel attention weight s of each channel. The activation operation can be implemented by the activation function of the fully connected neural network. The calculation formula of the activation operation can be expressed as the following formula (2): (2) In the above formula (2), z represents the spatial characteristics obtained by the extrusion operation. Let W represent the activation processing function for the activation operation, and let W represent the learnable dimension parameter. Let g represent the ReLU activation function, and g represent the channel attention function. represents the sigmoid activation function, Represents the dimensionality reduction parameter. This represents the dimension-up parameter, which is typically set to 16.

[0044] Here, it can be seen from formula (2) that when performing the excitation operation, the method of first reducing the dimensionality and then increasing the dimensionality for spatial features can greatly reduce the computational load of the network.

[0045] like Figure 4As shown, through the activation operation, the corresponding response weights, i.e., the channel attention weights s of each channel, can be determined, serving as the main indices for each channel in the feature map. Finally, a channel-weighted multiplication operation is performed, applying the channel attention weights s to the feature map of each channel in the convolutional feature map U. Weighted processing is performed to obtain the corresponding channel feature maps. The weighted calculation process is expressed as follows: (3) (3) In the above formula (3), This represents the weighted channel feature map. This represents the channel attention weight corresponding to the c-th channel. Let U represent the feature map of the c-th channel of the convolutional feature map U, and This indicates the feature calibration function used by the channel attention module for weighted calculation.

[0046] In this embodiment of the invention, a channel attention module is added to the enhanced generative model. By performing attention calculations on the channel feature maps in the convolutional feature maps, the adaptive adjustment of the feature responses of each channel can be achieved, and channel features that are not related to the extracted features can be suppressed, thereby reducing the computational load of the enhanced growth model and improving the model's feature extraction capability.

[0047] like Figure 4 As shown, after obtaining the channel feature map through the channel attention module, the enhanced generation model is then called to generate the generated image from the channel feature map. See here for more details. Figure 2 As shown, the EnlightenGAN model extracts channel feature maps, then uses deconvolution layers corresponding to the multi-scale dilated convolutional layers to generate the feature maps, thus restoring the feature maps to the image. In the U-Net structure, to avoid feature loss and generate more realistic images, the multi-scale dilated convolutional layers and deconvolutional layers are corresponding, and each layer is connected by an attention module. After the final deconvolutional layer outputs its feature map, it is then weighted and multiplied according to the attention map corresponding to the low-light image. The result of this weighted multiplication is then incorporated into the low-light image to obtain the generated image.

[0048] Of course, during the training of the enhanced generative model (EnlightenGAN model), an adversarial loss function is designed to minimize the distance between the actual light distribution and the output normal light distribution. Since a single global discriminator usually cannot exhibit adaptive ability when processing complex brightness images, a discriminator combining global and local discriminators is considered to enhance the adaptive ability and make each region of the enhanced image closer to natural light. A relative discriminator structure is used in the global discriminator to guide the generator network to generate more realistic false images. The standard functions of the relative discriminator are as follows: Equation (5) and Equation (6): (5) (6) In formulas (5) and (6) above, C represents the discriminator network. and They respectively represent following the true distribution and pseudo-distribution The sampled values, This represents the sigmoid function. This represents the expected value calculated after following a distribution.

[0049] If the least squares adversarial loss function is used instead of the sigmoid function, the resulting loss functions for the global discriminator D and generator G are expressed as follows: Equation (7) and Equation (8): (7) (8) In the above formulas (7) and (8), and They respectively represent following the true distribution and pseudo-distribution The sampled values, Let G represent the expected value calculated after following a distribution, where D is the global discriminator and G is the generator. The discriminator represents the relative discriminator.

[0050] For local discriminators, such as Figure 4 As shown, each time several small parts are randomly cropped from the low-light image and the generated image, the loss functions of the resulting local discriminator D and generator G are as follows: Equation (9) and Equation (10): (9) (10) In formulas (9) and (10) above, D is the local discriminator and G is the generator. and They respectively represent those that follow a local true distribution. and local pseudo-distribution The sampled values.

[0051] By using global and local loss functions, a global-local discriminator structure is employed during image enhancement to ensure that all local patches in the enhanced image appear as if exposed to natural light, thus avoiding local overexposure or underexposure. The global part uses the global loss function, while the local part uses the local loss function. Finally, the self-preserving feature function (LSFP) is used as the loss function in the enhanced image output stage to ensure regularization and preserve the features of the original image. The LSFP is expressed as follows (11): (11) In the above formula (11), This represents the input low-light image. This indicates that the generator G outputs an enhanced image. Let represent the feature map extracted from the low-light image using the VGG16 model trained on ImageNet, where i represents the i-th max-pooling layer and j represents the j-th convolutional layer after the i-th max-pooling. and These represent the dimensions of the extracted feature maps.

[0052] Finally, the global loss function, local loss function, and self-feature preservation loss function of the generator and discriminator are combined to train the augmentation generation model, enabling it to perform image augmentation on the acquired low-light images to be augmented, retaining more image feature points. This facilitates the extraction of more image feature points when locating the train later, thereby improving the train's positioning accuracy.

[0053] Step 103: Perform threshold segmentation on the generated image to obtain an enhanced image, and determine the keyframes from the enhanced image that are input into a preset sliding window.

[0054] In some embodiments, the main goal of the EnlightenGAN model is to enhance all low-resolution pixels in an image and reduce corresponding high-resolution pixels, thereby achieving image enhancement. However, the generated image may still exhibit local overexposure or underexposure, resulting in poor enhancement effects when applied to train images. Therefore, in this embodiment of the invention, threshold segmentation is performed on the generated image obtained by the EnlightenGAN model to obtain the enhanced image.

[0055] Specifically, for each pixel of the generated image, the pixel is thresholded by the improved threshold segmentation function T to obtain the segmented gray value of each pixel, thus obtaining the enhanced image. The threshold segmentation function T is expressed by the following formula (12): (12) In the above formula (12), This represents the grayscale value of a pixel (x, y). This represents the sigmoid function.

[0056] Furthermore, a first pixel threshold is preset for threshold segmentation. Second pixel threshold First pixel threshold Less than the second pixel threshold For each pixel in the generated image When the gray value of a pixel Less than the first pixel threshold At that time, the gray values ​​of the pixels are segmented. Set to 0; when the grayscale value of a pixel is... Greater than the first pixel threshold And less than the second pixel threshold At that time, the grayscale value of the pixel is... The ratio of 255 to 255 is used as the grayscale value of the pixel after segmentation. When the grayscale value of a pixel Greater than the second pixel threshold At that time, determine the grayscale value of the pixel. The ratio of 1 to 255 is used to determine the grayscale value of the pixel after segmentation. Therefore, the grayscale value after pixel segmentation It can be expressed as the following formula (13): (13) Finally, the grayscale values ​​are segmented based on pixel points. The enhanced image is determined. According to formula (13), the segmented gray value of each pixel after thresholding can be calculated, thus forming the segmented enhanced image.

[0057] In this embodiment of the invention, after the enhanced generation model generates an image, in order to make the generated image suitable for train image positioning, the threshold segmentation function in the enhanced generation model is improved to segment the generated image. This can effectively prevent local overexposure or underexposure that may occur after image enhancement, which is beneficial for subsequent train positioning.

[0058] The enhanced image after threshold segmentation can be used for train visual-inertial navigation localization. During train operation, visual-inertial odometry (VIO) can also measure the train's inertial measurement unit (IMU) data. To ensure good real-time performance of the VIO system, this embodiment of the invention employs a sliding window-based nonlinear optimization method to achieve tight coupling between VIO and IMU. The sliding window is preset and can be freely set according to actual conditions, consisting of several keyframes. Within the sliding window, the state variables are optimized through data fusion from VIO and IMU, i.e., the train's pose to be optimized is optimized. The sliding window does not iteratively optimize the pose at all times, but only estimates the pose within a few keyframes within the sliding window.

[0059] Since pose optimization is real-time, it requires continuously inputting corresponding keyframes into the sliding window. Therefore, after enhancing the captured low-light image to be enhanced to obtain the enhanced image, it is first necessary to determine the keyframes from the enhanced image to be input into the preset sliding window.

[0060] For each current frame in the enhanced image, if the current frame meets one of the criteria, then the current frame is determined as the key frame after the historical key frame in the preset sliding window. The historical key frame is the key frame that has been input into the sliding window. For example, the first frame of the enhanced image is directly input into the sliding window as the historical key frame. Then, the subsequent current frame is selected to enter the sliding window according to the criteria. The criteria include the following three, which are explained one by one below.

[0061] The first criterion is the number of image frames. It's essential to ensure that the number of image frames between the current frame and historical keyframes is greater than a preset frame threshold. It's important to note that you cannot simply select an image frame at regular intervals as a keyframe. This avoids selecting a large number of similar keyframes to fill the sliding window when the train is moving slowly, leading to a degradation in pose estimation. Conversely, when the camera is moving rapidly, many important image frames will be lost.

[0062] The second condition is the train's travel distance threshold. It is necessary to ensure that the train's travel distance between the current frame and historical keyframes is greater than a preset distance threshold. This ensures that the train has traveled a sufficiently long distance or undergone sufficient rotation between keyframes. In this embodiment of the invention, the degree of rotation is not constrained; it is only necessary to ensure that the train has traveled a sufficiently long distance. Considering that the time difference between keyframes is on the order of seconds, the distance threshold can be determined to be 15 meters based on the train's speed.

[0063] The second condition is that there are no shared feature points between keyframes. In other words, it must be ensured that the current frame and historical keyframes do not share any shared feature points. Shared feature points are those common to both the current frame and historical keyframes. Here, a new keyframe is only added when the camera leaves the current scene on the train, avoiding the problem of adding many image frames to the same scene.

[0064] Step 104: Within the sliding window, calculate the visual residual between feature points of key frames based on the feature points of the key frames, and calculate the inertial navigation error based on the train's inertial navigation data.

[0065] After determining the keyframes for the sliding window in step 103, to achieve tight coupling between VIO and IMU, pre-integration is performed on the IMU data acquired by the train's visual inertial navigation odometer. The IMU data includes triaxial acceleration and angular velocity data. For keyframes, Harris corner detection is used to extract image feature points, and multi-layer LK optical flow tracking is used to trace the extracted feature points from the Harris corner detection.

[0066] Next, based on the traditional visual-inertial navigation localization algorithm, a series of operations such as joint initialization, linear optimization, and edge detection are performed on the feature points of key frames and pre-integrated IMU data to estimate the train's pose in real time.

[0067] The sliding window method estimates the poses to be optimized only within a few keyframes within the sliding window. To maintain the size of the sliding window, each time a new keyframe is added, a certain keyframe within the sliding window needs to be removed. For removed keyframes, the state variables are no longer optimized, but the data cannot be discarded directly, as this would violate the original constraints. Therefore, a marginalization algorithm is used to transform the original constraints into a prior distribution, continuing to provide constraints on the state variables within the sliding window. This process is called sliding window marginalization, which is essentially a problem of obtaining a subset probability distribution of variables from a joint distribution. The marginalization operation ensures that the variables to be optimized in the sliding window do not continuously increase over time, guaranteeing the real-time performance of the system.

[0068] like Figure 5 As shown, the optimization process within the sliding window is essentially achieved through the inertial constraints of IMU pre-integration and the visual constraints of the visual residuals of key frame feature points. Therefore, within the sliding window, it is necessary to calculate the visual residuals of feature points between key frames based on the feature points of the key frames, and also to calculate the inertial navigation error based on the train's inertial navigation data.

[0069] The following describes the process of calculating the visual residual between keyframes based on their feature points. First, considering the varying richness of feature points within keyframes—because when a train passes through a tunnel, a portion of the low-light image captured by the camera sensor might contain the tunnel portion—the feature points in this tunnel portion exert a greater constraint on the keyframe compared to feature points in the rest of the image. Therefore, an adaptive search region is defined within the tunnel portion of the keyframe, and the calculation of the visual residual requires distinguishing between the inside and outside of this adaptive search region.

[0070] In this embodiment of the invention, the adaptive search area of ​​the tunnel portion of the keyframe is referred to as the region of interest (ROI). Therefore, when calculating the visual residual, it is necessary to determine the ROI in the keyframe. This is determined based on the position of the tunnel portion as captured by the camera sensor within the entire image. Generally, it is set to the middle third of the keyframe, that is, the image height is divided into three equal parts, and the middle third is taken as the ROI.

[0071] like Figure 5 As shown, within the sliding window, based on the region of interest set in the keyframe, the feature points are divided into feature points within the adaptive search area and feature points outside the adaptive search area, while constraints are applied to the keyframe.

[0072] When calculating the visual residual, it is divided into inner and outer parts. First, the inner reprojection error between keyframes is calculated based on the inner feature points within the region of interest (ROI), and the outer reprojection error is calculated based on the outer feature points outside the ROI. The inner and outer feature points are assigned different feature weights, which characterize the relative importance of constraints within and outside the ROI; generally, the feature weights of the inner feature points are greater than those of the outer feature points. Finally, the sum of the inner and outer reprojection errors is taken as the visual residual of the feature points between keyframes. The calculation formula is as follows: Formula (14): (14) In the above formula (14), , Let represent the set of all observed feature points within and outside the rectangular area of ​​the emotion line, respectively, and j represent the image of the j-th keyframe; Indicates the Each observed feature point. For robust kernel functions, This represents the total number of feature points on the j-th keyframe image. , represents the number of feature points within and outside the rectangle of interest in the j-th keyframe image, respectively. by These represent the feature weights set for the inner and outer feature points, respectively. The value range is (1, 2). Indicates the keyframe image of frame j as the keyframe of frame j. Observations of a feature point This indicates its corresponding weight. and X represents the reprojection error calculated by X, and X represents the state variable of the train, i.e. the pose to be optimized.

[0073] While calculating the visual residuals between keyframes within the sliding window, it is also necessary to calculate the inertial navigation error based on the train's inertial navigation data to provide inertial navigation constraints for the keyframes. For example... Figure 5 As shown, between keyframes, the corresponding IMU motion constraints need to be calculated based on the IMU data, which will be explained in detail below.

[0074] First, the inertial residuals are calculated based on the adjacent IMU measurements in the train's inertial navigation data and the pose to be optimized. Then, the marginalized prior residuals within the sliding window are obtained. And based on prior residuals Jacobian matrix Determine the position increment of the pose X to be optimized, and then use the difference between the prior residual and the position increment. The position error of the pose to be optimized is determined, and finally the sum of the inertial residual and the position error is taken as the inertial navigation error. It is expressed as the following formula (15): (15) In the above formula (15), B represents the set of measurements of all IMU inertial navigation data; k represents the measurement of the kth inertial navigation data. Represents the measurement value of IMU inertial navigation data Provided inertial residual, Indicates the measured value The corresponding weights, where X represents the pose to be optimized.

[0075] In this embodiment of the invention, a sliding window method is used for pose optimization. The corresponding visual and inertial navigation errors are calculated for key frames added in real time within the sliding window. At the same time, inertial and visual constraints are provided for the key frames, which ensures the real-time performance of train pose optimization and improves the accuracy of positioning.

[0076] Step 105: When the visual-inertial navigation error reaches its minimum value, determine the optimal pose estimate of the train.

[0077] The visual error is simultaneously calculated in step 104. and inertial navigation error Then, the two are combined, that is, the visual residual and the inertial navigation error are summed to obtain the visual-inertial navigation error. Optimizing the pose to be optimized is actually optimizing the visual-inertial navigation error. The smaller the visual-inertial navigation error, the more accurate the estimation of the pose to be optimized. Only when the visual-inertial navigation error reaches its minimum value can the optimal pose estimate of the train be determined.

[0078] Based on this, the present invention transforms the problem of estimating the pose to be optimized into the problem of solving the maximum a posteriori estimate, which can be solved by the least squares problem, and can be summarized as the following formula (16): (16) In the above formula (16), X represents the final optimal pose estimate, where X represents the pose to be optimized. This represents visual error, as detailed in formula (14) above. This represents the inertial navigation error, as detailed in formula (15) above.

[0079] After transforming the problem of estimating the pose to be optimized into solving the maximum a posteriori estimate problem, the estimation of the pose to be optimized can be transformed into the following formula: (16) In the above formula (16), x represents the pose to be optimized. To minimize the visual-inertial navigation error, it is necessary to calculate... The minimum value is obtained by iteratively calculating the minimum value of the visual-inertial navigation error within a sliding window by setting the descent step size of the train's pose to be optimized. The following describes the iterative calculation of the minimum value of the visual-inertial navigation error based on the incremental value of the pose to be optimized.

[0080] According to formulas (15) and (16), it can be seen that Since it is a nonlinear function, its minimum value cannot be calculated by differentiation. In this embodiment of the invention, an iterative method is used to solve it. First, the initial value of the pose to be optimized, x, is determined. Then proceed with the iteration. During each iteration, based on the descent step size... (k represents the increment value for the k-th iteration) Determine the initial value The visual-inertial navigation error is calculated based on the incremental value. The minimum value of the minimum value. To determine whether the current iteration has an effect, each iteration needs to calculate the gain ratio P of the minimum value, expressed as follows: (17) In the above formula (17), Let L represent the objective function for calculating the optimal estimate, and let L represent the second-order approximation function of the objective function.

[0081] When the gain ratio P at the minimum value is greater than 0, it indicates that the descent step size determined in this iteration is correct. (Taking negative values) can make the objective function If the descent allows for optimization of the pose to be optimized, then it is determined to proceed to the next iteration.

[0082] In order to make the objective function For rapid descent and convergence, this embodiment of the invention presets a gain threshold (greater than 0). After calculating the gain ratio at the minimum value in each iteration, it is determined whether the gain ratio exceeds the gain threshold. When the gain ratio is greater than the preset gain threshold, it indicates that the iteration is effective, which can accelerate the iteration pace and increase the increment value. The descent step size in the next iteration This allows the pose to be optimized to decrease rapidly after each iteration.

[0083] Through continuous iteration, the pose x to be optimized will decrease continuously, but it cannot decrease indefinitely to 0. Therefore, an incremental threshold is set here. When the incremental value decreases to the incremental threshold, it means that the incremental value is small enough, and the iteration stops. The minimum value of the current iteration is taken as the minimum value of the visual inertial navigation error.

[0084] When the increment value does not decrease to the increment threshold, the increment value will be... As the initial value of the pose to be optimized in the next iteration, and according to the value of the gain ratio, the descent step size of the increment value in the next iteration is increased.

[0085] As mentioned above, when the increment value drops to the increment threshold, the minimum value is reached. As the minimum value of visual-inertial navigation error, then the corresponding That confirms it. The method that minimizes the visual-inertial navigation error within the sliding window can be determined as the most accurate and optimal pose estimate for the train.

[0086] In this embodiment of the invention, the train pose estimation optimization problem is transformed into a problem of finding the minimum value, and the optimal pose estimation value of the train is determined when the visual inertial navigation error reaches the minimum value. This not only realizes the train pose estimation, but also ensures the accuracy of the pose estimation, thereby improving the positioning effect of the train.

[0087] In some embodiments, an iterative method is used to find the minimum value of the objective function, and the most critical point in this process is the selection of the descent step size. Determining the descent step size and selecting an appropriate increment value to ensure the objective function descends stably and quickly and converges is crucial in the solution process. Based on this, this embodiment of the invention employs the trust region dogleg method to select the descent step size, achieving a stable and rapid descent and convergence of the objective function. The process of determining the descent step size of the pose to be optimized using the trust region dogleg method is described in detail below.

[0088] First, the trust-based dog-leg algorithm is called to calculate the first descent step size under the steepest gradient method. The second descent step size under Gauss-Newton's method Based on the calculation process of the trust dogleg algorithm, a trust region was determined and set. Then, based on the relationship between the first descent step size, the second descent step size and the radius of the trust region, the result obtained by the trust dogleg algorithm was finally determined as the descent step size of the pose to be optimized.

[0089] like Figure 6 As shown, there are three possible relationships between the first descent step size, the second descent step size, and the trust region radius, which will be introduced one by one below.

[0090] The first type is as follows Figure 6 As shown in (1), when the second descent step size is not greater than the radius of the trust region set in the trust dogleg algorithm, that is, when the condition is met... At that time, the second descent step size will be... Directly used as the descent step size for the pose to be optimized .

[0091] The second type is as follows Figure 6 As shown in (2), when the first descent step size is not less than the radius of the trust region, that is, when the condition is met... At that time, determine the first descent step size. The unit step size value, where the unit step size is the first descent step size. Modulus of the first descent step The ratio of the unit step size value to the trust region radius is used. The product of these two factors is used as the descent step size for the pose to be optimized. .

[0092] The third type is as follows Figure 6 As shown in (3), the trust region radius is not less than the first descent step size and not greater than the second descent step size, that is, it satisfies... At that time, determine the first descent step size. With the second descent step size The step size difference is then used to preset the interpolation weights. The step size differences are weighted, and the weighted result is compared with the first descent step size. The sum of these values ​​is used as the descent step size for the pose to be optimized. .

[0093] Based on the above three relationships, the descent step size of the pose to be optimized It can be expressed as the following formula (18): (18) In the above formula (18), the interpolation weights The selection is based on the relationship between the increment value and the trust region radius. Therefore, in each iteration, the descent step size is calculated using the trust dogleg method combined with formula (18). As the basis for this iteration This is to determine the incremental value of the pose to be optimized, and ultimately achieve the objective function. The train descends and converges rapidly and steadily, achieving optimal pose estimation.

[0094] In this embodiment of the invention, by performing dilated convolution on the low-light image to be enhanced captured by the train, the receptive field of the network can be increased, more image context information can be fused, and channel attention can be calculated during the dilated convolution process to achieve adaptive adjustment of image channels and reduce the computational load of the model. Thresholding segmentation of the generated image can also overcome the defect of the enhancement generation model's effect of enhancing low pixels and weakening high pixels being ineffective in train localization. Finally, the enhanced image is used for vision / inertial navigation-based train localization, while simultaneously calculating the vision / inertial navigation error within the sliding window. This reduces the error while determining the optimal pose estimate, improving the train localization accuracy.

[0095] The following describes the urban rail transit train positioning device that integrates vision and inertial navigation provided by the present invention. The urban rail transit train positioning device that integrates vision and inertial navigation described below can be referred to in correspondence with the urban rail transit train positioning method that integrates vision and inertial navigation described above.

[0096] like Figure 7As shown, the urban rail transit train positioning device integrating vision / inertial navigation includes: extraction module 701, generation module 702, enhancement module 703, determination module 704, and positioning module 705. Specifically, the extraction module 701 is used to acquire a low-light image to be enhanced taken by the train, and call an enhancement generation model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps of the low-light image; the generation module 702 is used to process the convolutional feature maps extracted by each multi-scale dilated convolutional layer using a channel attention mechanism to obtain channel feature maps, and call the enhancement generation model to generate the channel feature maps to obtain a generated image; the enhancement module 703 is used to perform threshold segmentation processing on the generated image to obtain an enhanced image, and determine the key frames input into a preset sliding window from the enhanced image; the determination module 704 is used to calculate the visual reprojection error of feature points between key frames based on the feature points of the key frames within the sliding window, and calculate the inertial navigation error based on the train's inertial navigation data; the positioning module 705 is used to determine the optimal pose estimate of the train when the visual inertial navigation error reaches its minimum value, wherein the visual inertial navigation error is the sum of the visual reprojection error and the inertial navigation error.

[0097] It should be noted that the beneficial effects of the vision / inertial navigation integrated urban rail transit train positioning device mentioned above correspond to each other, so the beneficial effects of the vision / inertial navigation integrated urban rail transit train positioning device will not be elaborated here.

[0098] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a fusion vision / inertial navigation urban rail transit train positioning method. This method includes: acquiring a low-light image to be enhanced captured by the train; calling an enhancement generative model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps from the low-light image; performing channel attention processing on the convolutional feature maps extracted by each multi-scale dilated convolutional layer to obtain channel feature maps, and calling the enhancement generative model to generate a generated image from the channel feature maps; performing threshold segmentation processing on the generated image to obtain an enhanced image, and determining keyframes from the enhanced image that are input into a preset sliding window; within the sliding window, calculating the visual residual between feature points of the keyframes based on the feature points of the keyframes, and calculating the inertial navigation error based on the train's inertial navigation data; when the visual-inertial navigation error reaches its minimum value, determining the optimal pose estimate of the train, wherein the visual-inertial navigation error is the sum of the visual residual and the inertial navigation error.

[0099] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the vision / inertial navigation fusion urban rail transit train positioning method provided by the above methods. The method includes: acquiring a low-light image to be enhanced taken by the train; calling an enhancement generative model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps of the low-light image; performing channel attention mechanism processing on the convolutional feature maps extracted by each multi-scale dilated convolutional layer to obtain channel feature maps, and calling the enhancement generative model to generate the channel feature maps to obtain a generated image; performing threshold segmentation processing on the generated image to obtain an enhanced image, and determining keyframes from the enhanced image that are input into a preset sliding window; within the sliding window, calculating the visual residual between feature points of the keyframes based on the feature points of the keyframes, and calculating the inertial navigation error based on the train's inertial navigation data; when the visual inertial navigation error reaches its minimum value, determining the optimal pose estimate of the train, wherein the visual inertial navigation error is the sum of the visual residual and the inertial navigation error.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for fusion vision / inertial navigation urban rail transit train positioning provided by the methods described above. This method includes: acquiring a low-light image to be enhanced captured by the train; calling an enhancement generative model with multi-scale dilated convolutional layers to extract multiple convolutional feature maps from the low-light image; performing channel attention processing on the convolutional feature maps extracted by each multi-scale dilated convolutional layer to obtain channel feature maps, and calling the enhancement generative model to generate a generated image from the channel feature maps; performing threshold segmentation processing on the generated image to obtain an enhanced image, and determining keyframes from the enhanced image that are input into a preset sliding window; within the sliding window, calculating the visual residual between feature points of the keyframes based on the feature points of the keyframes, and calculating the inertial navigation error based on the train's inertial navigation data; when the visual inertial navigation error reaches its minimum value, determining the optimal pose estimate of the train, wherein the visual inertial navigation error is the sum of the visual residual and the inertial navigation error.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0103] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for positioning urban rail transit trains integrating vision / inertial navigation, characterized in that: The method comprises: Obtain a low-light image to be enhanced taken by a train, and call an enhancement generation model with a multi-scale hole convolution layer to extract multiple convolution feature maps of the low-light image; Performing channel attention mechanism processing on the convolution feature map extracted by each multi-scale hole convolution layer to obtain a channel feature map, and calling the enhanced generation model to generate the channel feature map to obtain a generated image; Performing threshold segmentation processing on the generated image to obtain an enhanced image, and determining a key frame input into a preset sliding window from the enhanced image; In the sliding window, the visual residual of the feature points between the key frames is calculated according to the feature points of the key frames, and the inertial navigation error is calculated according to the inertial navigation data of the train; When the visual inertial navigation error reaches a minimum value, an optimal pose estimation value of the train is determined, wherein the visual inertial navigation error is the sum of the visual residual and the inertial navigation error.

2. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The calling of the enhanced generation model with a multi-scale hole convolution layer to extract the convolution feature map of the low-light image includes: Performing dilated convolution processing on the low-light image in sequence by enhancing multiple multi-scale dilated convolution layers of the generation model to obtain a convolution feature map output by each multi-scale dilated convolution layer; Among them, the dilated convolution processing process of the multi-scale dilated convolution layer includes: Calling a convolution network with a convolution kernel of 1×1 to perform convolution processing on the low-light image to obtain a first input feature map; Calling multiple dilated convolution networks with different linear expansion factors to perform dilated convolution processing on the low-light image to obtain multiple second input feature maps; The first input feature map is stacked with each of the second input feature maps respectively, and the stacked feature maps are input into a convolution network with a convolution kernel of 5×5 for convolution processing to obtain a new feature map; The new feature map and the low-light image are input into a convolution network with a convolution kernel of 1×1 for convolution processing to obtain a convolution feature map.

3. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The convolution feature map extracted by each multi-scale hole convolution layer is processed by a channel attention mechanism to obtain a channel feature map, including: The convolution feature map extracted by each multi-scale hole convolution layer is squeezed to obtain the spatial features of each layer of the corresponding channel; Performing an excitation operation on the spatial features to obtain a channel attention weight of each layer of channels; The feature map of each layer of the channel in the convolution feature map is weighted by the channel attention weight to obtain a corresponding channel feature map.

4. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The step of performing threshold segmentation processing on the generated image to obtain an enhanced image includes: Presetting a first pixel threshold and a second pixel threshold for threshold segmentation, wherein the first pixel threshold is smaller than the second pixel threshold; For each pixel in the generated image, when the grayscale value of the pixel is less than the first pixel threshold, setting the grayscale value of the pixel after segmentation to 0; When the grayscale value of the pixel is greater than the first pixel threshold and less than the second pixel threshold, the ratio of the grayscale value of the pixel to 255 is used as the grayscale value of the pixel after segmentation; When the grayscale value of the pixel is greater than the second pixel threshold, determining a ratio of the grayscale value of the pixel to 255, and determining a difference between 1 and the ratio as a grayscale value of the pixel after segmentation; An enhanced image is determined based on the segmented grayscale values ​​of the pixels.

5. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The step of determining a key frame input into a preset sliding window from the enhanced image comprises: For each current frame in the enhanced image, if the current frame satisfies one of the conditions, the current frame is determined as a key frame after a historical key frame in a preset sliding window; The conditions include: The number of image frames between the current frame and the historical key frame is greater than a preset frame number threshold; The train running distance between the current frame and the historical key frame is greater than a preset distance threshold; There is no common view feature point between the current frame and the historical key frame, and the common view feature point is a common feature point between the current frame and the historical key frame.

6. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The calculating the visual residual of the feature points between the key frames according to the feature points of the key frames comprises: Determining a rectangular region of interest in the key frame; Calculating an inner reprojection error between key frames according to inner feature points within the rectangular region of interest, and calculating an outer reprojection error between key frames according to outer feature points outside the rectangular region of interest, wherein the inner feature points and the outer feature points are provided with different feature weights; The sum of the inner reprojection error and the outer reprojection error is used as the visual residual of the feature points between key frames.

7. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The calculating the inertial navigation error according to the inertial navigation data of the train includes: Calculate the corresponding inertial residual according to the adjacent IMU measurement values ​​in the train's inertial navigation data and the posture to be optimized; Obtaining a marginalized prior residual in the sliding window, and determining a position increment of the pose to be optimized based on a Jacobian matrix of the prior residual; Determining the position error of the pose to be optimized according to the difference between the prior residual and the position increment; The sum of the inertial residual and the position error is taken as the inertial guidance error.

8. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 1 is characterized in that: The minimum value of the visual inertial navigation error is obtained by iteratively calculating after setting the descending step length of the train's posture to be optimized in the sliding window, and iteratively calculating the minimum value of the visual inertial navigation error according to the descending step length of the posture to be optimized, including: Determine the initial value of the pose to be optimized; In each iteration process, an incremental value of the initial value is determined according to the descending step length, and a minimum value of the visual inertial navigation error is calculated according to the incremental value; When the increment value drops to the increment threshold, the iteration is stopped, and the minimum value is taken as the minimum value of the visual inertial navigation error; When the incremental value does not drop to the incremental threshold, the incremental value is used as the initial value of the posture to be optimized in the next iteration. When the gain ratio of the minimum value is greater than 0, it is determined to enter the next iteration. When the gain ratio is greater than the preset gain threshold, the decreasing step size of the incremental value in the next round of iteration is increased.

9. The urban rail transit train positioning method integrating vision / inertial navigation according to claim 8 is characterized in that: The process of determining the descending step length of the posture to be optimized includes: The trust dogleg algorithm is called to calculate the first descent step length under the fastest gradient method and the second descent step length under the Gauss-Newton method; When the second descent step length is not greater than the trust region radius set in the trust dogleg algorithm, the second descent step length is used as the descent step length of the pose to be optimized; When the first descent step length is not less than the trust region radius, determining a unit step length value of the first descent step length, and taking the product of the unit step length value and the trust region radius as the descent step length of the posture to be optimized; When the trust region radius is not less than the first descent step length and not greater than the second descent step length, determining a step length difference between the first descent step length and the second descent step length; The step length difference is weighted by a preset interpolation weight, and the weighted result is added to the first descent step length as the descent step length of the posture to be optimized.

10. A urban rail transit train positioning device integrating vision / inertial navigation, characterized in that: The device comprises: An extraction module is used to obtain a low-light image to be enhanced taken by a train, and call an enhancement generation model with a multi-scale hole convolution layer to extract multiple convolution feature maps of the low-light image; A generation module, used for performing channel attention mechanism processing on the convolution feature map extracted by each multi-scale hole convolution layer to obtain a channel feature map, and calling the enhanced generation model to generate the channel feature map to obtain a generated image; An enhancement module, used for performing a threshold segmentation process on the generated image to obtain an enhanced image, and determining a key frame input into a preset sliding window from the enhanced image; A determination module, configured to calculate, within the sliding window, a visual reprojection error of feature points between key frames based on feature points of the key frames, and calculate an inertial navigation error based on inertial navigation data of the train; The positioning module is used to determine the optimal pose estimation value of the train when the visual inertial navigation error reaches a minimum value, and the visual inertial navigation error is the sum of the visual reprojection error and the inertial navigation error.

Images