A visual marker robust positioning method and system for complex lighting environment

Abstract

The application discloses a kind of visual mark robust positioning method and system for complex illumination environment, belong to robot positioning technical field, its technical scheme main point is including, obtain the current corresponding multi-source data set of inspection robot and carry out pre-processing, multi-source data set includes RGBD data and point cloud data;According to the laser composite feature matrix obtained according to pre-processed point cloud data;According to the visual composite feature matrix obtained according to pre-processed RGBD data and residual network model;According to laser composite feature matrix and visual composite feature matrix, obtain fusion matrix;According to the position of the current inspection robot where according to the fusion matrix corresponding to adjacent last time fusion matrix, the present application extracts visual composite feature matrix by means of residual network model, based on laser composite feature matrix, the geometric structure information of environment is completely retained, through the fusion of laser and visual composite feature matrix, improve the position perception accuracy of inspection robot in complex scene.

Description

Technical Field

[0001] This invention relates to the field of robot localization technology, and more specifically to a visual marker-based robust localization method and system for complex lighting environments. Background Technology

[0002] In smart agricultural factories, inspection robots need to perform all-weather tasks such as autonomous navigation, crop phenotypic data collection, and equipment status monitoring. However, the complexity of the actual operating environment brings many challenges to traditional positioning technologies. For example, the feature representation capabilities of a single data source have inherent limitations. They are easily affected by drastic changes in lighting, object occlusion, glass / metal reflections, etc., which can lead to visual feature distortion and loss of details, thus causing positioning drift. Furthermore, the robustness of geometric features in dynamic scenes is insufficient, making traditional positioning solutions prone to failure. Therefore, existing technologies have shortcomings. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a robust visual marker localization method and system for complex lighting environments. By fusing visual and laser features, the complementary advantages of the two types of features are fully utilized, thereby improving the robustness and comprehensiveness of environmental features.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] This invention provides a robust visual marker localization method for complex lighting environments, comprising:

[0006] The multi-source dataset corresponding to the inspection robot is acquired and preprocessed. The multi-source dataset includes RGBD data and point cloud data.

[0007] The laser composite feature matrix is ​​obtained from the preprocessed point cloud data;

[0008] Based on the preprocessed RGBD data and the residual network model, the visual composite feature matrix is ​​obtained;

[0009] Based on the laser composite feature matrix and the visual composite feature matrix, a fusion matrix is ​​obtained;

[0010] The current position of the inspection robot is obtained based on the fusion matrix and the fusion matrix corresponding to the previous time step.

[0011] As a further improvement of the present invention, the step of obtaining the laser composite feature matrix based on the preprocessed point cloud data includes:

[0012] The point cloud data is divided into multiple grids, and each grid includes multiple data points;

[0013] The normal and curvature corresponding to each grid are obtained based on the data points;

[0014] The laser composite feature matrix is ​​obtained by mapping the normals and curvatures corresponding to each grid and the fully connected layer.

[0015] As a further improvement of the present invention, the step of obtaining the visual composite feature matrix based on the preprocessed RGBD data and the residual network model includes:

[0016] Based on the preprocessed RGBD data and the residual network model, shallow features, mid-level features, deep features, and depth multi-scale features are obtained.

[0017] Calculate the confidence scores for the shallow features, mid-level features, deep features, and depth multi-scale features respectively;

[0018] The visual composite feature matrix is ​​obtained based on the confidence score and the LSS inverse deep network.

[0019] As a further improvement of the present invention, the step of obtaining the visual composite feature matrix based on the confidence score and the LSS inverse deep network includes:

[0020] Initialize the weight vector and update each element in the initial weight vector according to the confidence score to obtain the optimal weight vector;

[0021] The shallow features, mid-level features, deep features, and deep multi-scale features are fused according to the optimal weight vector to obtain fused multi-scale features;

[0022] Based on the fused multi-scale features and the LSS inverse depth network, a visual composite feature matrix is ​​obtained.

[0023] As a further improvement of the present invention, the step of updating each element in the initial weight vector according to the confidence score to obtain the optimal weight vector includes:

[0024] The current reflectance intensity is obtained based on the multi-source dataset;

[0025] Based on the current reflectivity and empirical parameter settings, the reflectivity correction factor is obtained;

[0026] The optimal weight vector is obtained by updating each element in the initial weight vector based on the reflection correction factor and the confidence score.

[0027] As a further improvement of the present invention, the step of updating each element in the initial weight vector according to the reflectivity correction factor and the confidence score to obtain the optimal weight vector includes:

[0028] The current weight vector is obtained by updating each element in the initial weight vector according to the reflection correction factor and the confidence score.

[0029] Based on the current weight vector, the shallow features, mid-level features, deep features, and deep multi-scale features are fused to obtain the current fused multi-scale features;

[0030] Perform a first iteration operation on the current fused multi-scale features. The first iteration operation includes obtaining the feature response intensity and label matching success rate based on the current fused multi-scale features, obtaining the overall confidence based on the feature response intensity and label matching success rate, determining whether the overall confidence reaches a preset termination condition, and if not, updating the current weight vector and the current fused multi-scale features according to the scaling factor until the preset termination condition is reached, and outputting the optimal weight vector.

[0031] As a further improvement of the present invention, the step of obtaining the fusion matrix based on the laser composite feature matrix and the visual composite feature matrix includes:

[0032] Align the laser composite feature matrix and the visual composite feature matrix;

[0033] The current fusion matrix is ​​obtained based on the aligned laser composite feature matrix and visual composite feature matrix;

[0034] The second iteration operation is performed based on the current fusion matrix. The second iteration operation includes determining the misalignment region based on the current fusion matrix, obtaining the translation correction matrix based on the query features corresponding to each pixel in the misalignment region, updating the current fusion matrix based on the translation correction matrix, until the current fusion matrix converges, and outputting the fusion matrix.

[0035] As a further improvement of the present invention, a translation correction matrix is ​​obtained based on the query features corresponding to each pixel in the misalignment region, including:

[0036] The weight corresponding to each pixel in the misalignment area is determined based on the binarized reflective area mask, wherein the binarized reflective area mask is obtained based on the multi-source dataset;

[0037] The query feature is obtained based on the weight corresponding to each pixel in the misalignment region;

[0038] Based on the query features, obtain the matching corresponding point for each pixel in the misalignment region;

[0039] Construct an error function based on the matching points;

[0040] The translation correction matrix is ​​obtained by solving the error function using the least squares method.

[0041] As a further improvement of the present invention, the step of obtaining the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step includes:

[0042] The unstable region is determined based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step.

[0043] The fusion matrix is ​​updated based on the unstable region;

[0044] The initial pose is obtained based on the updated fusion matrix;

[0045] The current position of the inspection robot is obtained based on the initial pose and the sliding window.

[0046] This invention provides a robust visual marker localization system for complex lighting environments, comprising:

[0047] The data acquisition module is used to acquire and preprocess the multi-source dataset currently corresponding to the inspection robot. The multi-source dataset includes RGBD data and point cloud data.

[0048] The calculation module is used to obtain the laser composite feature matrix based on the preprocessed point cloud data; and to obtain the visual composite feature matrix based on the preprocessed RGBD data and the residual network model.

[0049] The fusion module is used to obtain a fusion matrix based on the laser composite feature matrix and the visual composite feature matrix;

[0050] The positioning module is used to obtain the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the previous adjacent time step.

[0051] This invention utilizes a residual network model to extract a visual composite feature matrix, enhancing the deep semantic expression and detail representation capabilities of visual features. Simultaneously, a laser composite feature matrix constructed based on point cloud data fully preserves the geometric structure information of the environment. By fusing the laser and visual composite feature matrices, the complementary advantages of the two types of features are fully leveraged, improving the robustness and comprehensiveness of environmental features. Finally, localization is achieved by matching the current and previous fused matrices, effectively solving the problem of insufficient feature representation from a single data source and significantly improving the position perception accuracy and stability of the inspection robot in complex scenarios. Attached Figure Description

[0052] Figure 1 A schematic diagram illustrating the steps of a robust visual marker localization method for complex lighting environments provided by this invention;

[0053] Figure 2 This is a schematic diagram of the structure of the residual network model provided by the present invention;

[0054] Figure 3 This invention provides a schematic diagram of the steps for obtaining fused multi-scale features.

[0055] Figure 4 This is a schematic diagram illustrating the steps for obtaining the fusion matrix provided by the present invention. Detailed Implementation

[0056] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof.

[0057] The term "and / or" in the following text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0058] like Figure 1 As shown, this application provides a robust visual marker localization method for complex lighting environments, including:

[0059] The inspection robot's current multi-source dataset is acquired and preprocessed. The multi-source dataset includes RGBD data and point cloud data.

[0060] The laser composite feature matrix is ​​obtained from the preprocessed point cloud data;

[0061] Based on the preprocessed RGBD data and the residual network model, the visual composite feature matrix is ​​obtained;

[0062] The fusion matrix is ​​obtained based on the laser composite feature matrix and the visual composite feature matrix;

[0063] The current position of the inspection robot can be obtained by using the fusion matrix and the fusion matrix corresponding to the previous time step.

[0064] This embodiment is applied to the positioning of inspection robots in a smart agricultural factory. Specifically, firstly, multi-source datasets are collected through sensors. For example, the sensors in this embodiment include RGBD cameras, LiDAR, reflective sensors, and auxiliary sensors. This embodiment does not limit the specific installation position of the sensors on the inspection robot; those skilled in the art can determine it according to the actual situation. For example, the reflective sensor can be installed on the top of the inspection robot to face the greenhouse glass. When the inspection robot reaches a certain position, the RGBD camera acquires a color image of the scene in front of it and emits infrared lasers. The distance corresponding to each pixel in the color image is obtained based on the round-trip time of the laser, thus obtaining the depth map corresponding to the current scene. The infrared sensor configured with RGBD captures the infrared reflection signal separately to obtain the infrared map corresponding to the current scene. The set of images obtained by the RGBD camera is RGBD data. Through the multiple lasers emitted by the LiDAR, the three-dimensional point cloud data of 360° around the current inspection robot can be obtained, and the round-trip time and reflected light intensity corresponding to each laser beam can be obtained. The reflective sensor is used to collect the intensity of ambient light reflected from objects in front. Auxiliary sensors include a light sensor, a temperature and humidity sensor, and an IMU. The light sensor is used to obtain the total light intensity of the current environment based on the built-in photoresistor. The proportion of the intensity of reflected ambient light in the total light intensity of the current environment is recorded as the reflective intensity. Preferably, since the detection area corresponding to the reflective sensor is smaller than the area corresponding to the color image, multiple reflective sensors can be installed to obtain the intensity of ambient light reflected from different objects in multiple directions in order to obtain more accurate and comprehensive data. When calculating the reflective intensity, the movement direction of the inspection robot can be determined first. Different weights are set for the intensity of ambient light reflected in different directions according to the movement direction. This embodiment does not limit the specific value of the weight. The closer it is to the movement direction, the greater the weight of the ambient light intensity reflected in that direction. Finally, the weighted value and its proportion in the total light intensity of the current environment are recorded as the reflective intensity. The IMU is used to obtain the quaternion corresponding to the current posture of the inspection robot. Each sensor in this embodiment is an existing sensor, so the working principle of the sensor will not be described in detail in this embodiment.

[0065] Afterwards, data preprocessing is performed to obtain preprocessed RGBD data and preprocessed point cloud data. For example, the lighting scene category corresponding to the current scene is first determined based on the total light intensity of the current environment collected by the light sensor. This embodiment does not limit the classification method of the light sensor. For example, it can be divided into overexposure with supplementary lighting, backlight, weak light, and normal lighting. This embodiment does not limit the specific value of the total light intensity of the current environment corresponding to each category. Those skilled in the art can determine it according to the actual situation. For example, when the total light intensity of the current environment is greater than 50,000 lux, the lighting scene category is overexposure with supplementary lighting. After determining the category corresponding to the lighting scene, the acquired color images are preprocessed according to different categories to obtain preprocessed color images. For example, when there is overexposure due to supplemental lighting, the Gamma correction method can be used to reduce the brightness of overexposed pixels by compressing the dynamic range of the highlight area, while brightening the details in the shadow area. In backlight, since the illumination component of the image dominates the visual information, while the reflection component, such as the real texture of crops, is masked, the Retinex algorithm can be used to decompose the image into illumination and reflection components. By suppressing drastic changes in the illumination component (such as overexposure in bright areas under backlight), the details of the reflection component (such as the marking texture in dark areas) are preserved. In low light, the image has problems with low contrast and high noise. Similarly, the Retinex algorithm can be used to enhance the contrast and Gaussian filtering can be applied to smooth the noise. In normal lighting, only mean filtering can be performed to avoid overprocessing that leads to the loss of original features. The above preprocessing algorithms are technical means that can be implemented by those skilled in the art, and this embodiment will not elaborate on them. Furthermore, this embodiment does not limit the parameters in the above algorithms, such as the size of the filtering window, which can be determined by those skilled in the art according to the actual situation.

[0066] For example, after obtaining the preprocessed color image, the brightness value corresponding to each pixel is compared with a preset brightness value. The area corresponding to the pixel with a brightness value greater than the preset brightness value is recorded as the first reflective area. The brightness value can be represented by a grayscale value. In this embodiment, the specific value of the preset brightness value is not limited. It can be obtained by statistically analyzing the brightness of reflective areas (such as glass) and normal walls in the factory. For example, it can be set to 220. Next, the intensity of the reflected ambient light collected by each reflective sensor is obtained, and the reflective intensity corresponding to each reflective sensor is obtained according to the total light intensity of the current environment. Then, the reflective intensity corresponding to each reflective sensor is compared with the light intensity threshold to determine the reflective sensor whose intensity is greater than the light intensity threshold. In this embodiment, the specific value of the light intensity threshold is not limited. For example, the light intensity threshold can be obtained based on the actual measurement data of different objects (such as glass, crop leaves, etc.). A value that can effectively distinguish reflective objects (such as glass) and crops, such as 60%, is selected as the light intensity threshold. Then, the detection area corresponding to the reflective sensor is determined to be the area corresponding to the preprocessed color image. This area is recorded as the second reflective area. Finally, the overlapping part of the first reflective area and the second reflective area is recorded as the reflective area. The remaining area in the preprocessed color image is recorded as the non-reflective area. Then, a binary reflective area mask is generated based on the reflective area and the non-reflective area. The size of the binary reflective area mask is the same as the size of the preprocessed color image, where the value corresponding to the reflective area is 1 and the value corresponding to the non-reflective area is 0.

[0067] Next, the depth map corresponding to the current scene is preprocessed to remove noise and compensate for missing values. This embodiment does not limit the specific preprocessing method; for example, noise can be removed by filtering algorithms and missing values ​​can be supplemented by interpolation. Then, the infrared image is preprocessed. Specifically, under normal lighting conditions, since the color image is clear enough, no infrared image assistance is needed, and therefore no preprocessing of the infrared image is required. In other types of lighting, the color image is affected by reflection, but objects causing reflection, such as glass, have low reflectivity to infrared light, so infrared light is not affected by reflection. This embodiment preprocesses the infrared image to assist in obtaining more accurate positioning results. For example, firstly, the infrared image can be divided into foreground and background based on the pixel value of each pixel in the infrared image using a preset threshold. The foreground can be a high diffuse reflectance visual marker affixed to the planting rack. This marker can use an infrared absorbing coating to facilitate... Bright patches appear in the infrared image, and the high diffuse reflectance visual markers serve as the core anchor points for localization in this application. This embodiment does not limit the shape and texture of these markers; for example, they can be combinations of color blocks with specific geometric shapes to facilitate subsequent multi-scale feature extraction steps. The background includes crops, walls, etc., materials with low infrared reflectivity, resulting in darker colors in the infrared image. The pixel values ​​corresponding to the foreground are then modified to 255, and the pixel values ​​corresponding to the background are modified to 0, thus obtaining the infrared image of the marker candidate region. That is, the infrared image of the marker candidate region only contains black and white colors. This embodiment does not limit the preset threshold, such as setting it to 120. Finally, preprocessed RGBD data is obtained, including the preprocessed color image, the preprocessed depth map, the infrared image of the marker candidate region, and a binarized reflective area mask.

[0068] This embodiment utilizes a residual network model to extract a visual composite feature matrix, enhancing the deep semantic expression and detail representation capabilities of visual features. Simultaneously, a laser composite feature matrix constructed based on point cloud data fully preserves the geometric structure information of the environment. By fusing the laser and visual composite feature matrices, the complementary advantages of the two types of features are fully utilized, improving the robustness and comprehensiveness of environmental features. Finally, localization is achieved by matching the current and previous fused matrices, effectively solving the problem of insufficient feature representation from a single data source and significantly improving the position perception accuracy and stability of the inspection robot in complex scenarios.

[0069] Furthermore, this embodiment provides a step for obtaining a laser composite feature matrix based on preprocessed point cloud data, including:

[0070] The point cloud data is divided into multiple grids, and each grid includes multiple data points;

[0071] The normal and curvature of each grid are obtained based on the data points;

[0072] The laser composite feature matrix is ​​obtained by mapping the normals and curvatures of each grid and the fully connected layer.

[0073] After obtaining point cloud data via LiDAR, preprocessing is required to remove invalid point cloud data, resulting in preprocessed point cloud data. Invalid point cloud data includes data points close to the ground and data points far away. The ground is a redundant background, and data points far away have larger errors and therefore need to be removed. This embodiment does not limit the criteria for determining whether a point is close to the ground or far from the inspection robot. Next, the actual area covered by the preprocessed point cloud data is divided into multiple grids. This embodiment does not limit the number of grids; for example, it can be divided into 200×200 grids. Each grid includes multiple data points from the point cloud data. Then, the normal and curvature corresponding to each grid are calculated. For example, for a grid, the centroids of its multiple data points are first calculated, and the covariance matrix corresponding to these multiple data points is calculated. Eigenvalue decomposition is performed on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors. The normal is the eigenvector corresponding to the smallest eigenvalue. Then, the sum of multiple eigenvalues ​​is calculated, and the ratio of the smallest eigenvalue to the sum of these multiple eigenvalues ​​is recorded as the curvature.

[0074] For each grid, after obtaining its corresponding normal and curvature, its corresponding data points, average height, and point cloud density can also be obtained. Then, the normal, curvature, average height, and point cloud density are normalized and input into the fully connected layer. The fully connected layer can map the initial low-dimensional features into multi-dimensional features to solve the problem of insufficient expressive power of low-dimensional features and make them adaptable to the structural requirements of subsequent neural networks. In this embodiment, the specific dimension after mapping is not limited. For example, each grid is mapped to a 128-dimensional vector. Then, the vectors obtained from mapping each grid are combined to obtain the initial feature matrix. Then, for each grid, the average flight time, average reflection intensity, and average propagation distance of its included data points are calculated. Obtaining the flight time, reflection intensity, and propagation distance of each data point based on lidar is a technical means that can be implemented by those skilled in the art, and this embodiment will not elaborate on this. After normalizing the average flight time, average reflection intensity, and average propagation distance, the values ​​are appended to the vector corresponding to each grid. Then, the laser composite feature matrix is ​​obtained based on the initial feature matrix. For example, assuming that a total of 200×200 grids are obtained, the initial feature matrix should be a tensor with a corresponding dimension of 200×200×128. The laser composite feature matrix after adding features is also a tensor with a corresponding dimension of 200×200×131. Since the normal, curvature, average height, and point cloud density are spatial geometric features, while the average flight time, average reflection intensity, and average propagation distance are sensor observation features, the two features are heterogeneous features with different attributes. Direct mapping will result in a large error. Therefore, this embodiment processes them separately.

[0075] Furthermore, this embodiment provides a step for obtaining a visual composite feature matrix based on preprocessed RGBD data and a residual network model, including:

[0076] Shallow features, mid-level features, deep features, and depth multi-scale features are obtained based on the preprocessed RGBD data and the residual network model.

[0077] Calculate the confidence scores for shallow features, mid-level features, deep features, and deep multi-scale features respectively;

[0078] Based on the confidence score and the LSS inverse deep network, the visual composite feature matrix is ​​obtained.

[0079] Specifically, after obtaining the preprocessed RGBD data, the preprocessed color image and the preprocessed depth map are first sized and stitched together. Since the acquired images may have reflections or occlusions, this embodiment stitches the depth map to supplement spatial contour information and improve robustness. The stitched result is then input into a residual network model to generate multi-scale features. For example, the ResNet-50 network model can be used. Based on the advantages of the residual structure, the residual network model can extract relatively stable features in scenes with reflections or occlusions. For instance, even if the image is overexposed in a reflective scene, the extracted deep semantic features can still identify the surrounding object structure through contextual information, avoiding complete failure of visual features. For example, such as... Figure 2As shown, the residual network model includes an input layer, pooling layers, and a deep branch network. Between the pooling layers and the deep branch network are multiple stages, each a feature extraction stage composed of stacked residual blocks. The core function of each stage is to progressively reduce the feature map size and increase the number of channels, thereby gradually obtaining deeper features. For example, after the concatenated result is input into the residual network model, it first passes through the input layer and pooling layer to extract the most basic low-level features, preserving details such as edges, colors, and contours of the original image. Assuming the pooling layer output has dimensions of 480×270×64, where 4... The input size is 80×270, and the pooling layer has 64 channels. The data then enters Stage 1, where convolution enhances the continuity of low-level features, making the edges and textures of visual markers clearer. This results in the output of shallow features, which can be 240×135×64 in dimension. Compared to the pooling layer output, the size is halved, while the number of channels remains the same. These shallow features are then input into Stage 2 to extract mid-level information such as shape and texture. This helps in locating visual markers when they are occluded by crops, using the continuity of crop texture to aid in localization. This outputs mid-level features, which can be 120×68×128 in dimension. The output of Stage 1 is halved in size and doubled in number of channels. The mid-level features are then input into Stage 3 to extract deep semantic features, which associate visual markers with the structure of the planting rack, resolving feature ambiguity in complex scenes. This results in the output of deep features with dimensions of 60×34×256, half the size and doubled in number of channels compared to the output of Stage 2. A deep branch network further processes the preprocessed depth map through residual blocks and convolutional layers to obtain multi-scale depth features. These multi-scale features directly reflect the spatial contours of objects, such as visual markers being... Even when an object obstructs the view, the depth contour can still locate the marked position. This embodiment does not limit the dimension of the depth multi-scale features. Preferably, to facilitate the subsequent generation of the visual composite feature matrix via the LSS inverse depth network, and to ensure consistency with the subsequent laser composite feature matrix, the size can be determined by referring to the grid number of the point cloud data. Since shallow features correspond to edge information, and depth multi-scale features correspond to spatial contour information, and edges and contours are related information, the number of channels corresponding to the depth multi-scale features is set to be the same as the number of channels corresponding to the shallow features. For example, the dimension of the depth multi-scale features is 200×200×64. This embodiment does not limit the number of stages or the dimension of the features; the above values ​​are merely examples to visually illustrate the process of continuously shrinking size and increasing channel number. Those skilled in the art can set specific values ​​according to actual conditions, such as increasing the number of stages when computing power is sufficient and more accurate results are required.

[0080] The residual network model is trained. For example, a training dataset is first constructed, collecting RGBD data from inspection scenarios, covering typical interference scenarios such as overexposure due to supplemental lighting, backlighting, low light, reflection, and crop occlusion. It also includes samples with high diffuse reflection visual markers. The data is then labeled to clarify the edges, textures, semantic structures, and spatial contour regions of the visual markers in each sample, serving as a reference for feature supervision. The RGBD data is then preprocessed according to the above steps to obtain the model's input data, which includes a training set and a validation set. The training set data is input into the model in batches, and the parameters in the model are iteratively updated using a loss function and backpropagation until a preset number of iterations is reached. The loss function is determined by multiple sub-losses. Each sub-loss corresponds to a feature. For example, the sub-loss corresponding to shallow features is the mean squared error between shallow features and true edge features; the sub-loss corresponding to mid-layer features is the mean squared error between mid-layer features and true texture features; the sub-loss corresponding to deep features is the mean squared error between deep features and true semantic structure features; and the sub-loss corresponding to deep multi-scale features is the mean squared error between deep multi-scale features and features corresponding to true spatial contours. Then, each sub-loss is weighted to obtain the loss function. When calculating the mean squared error, it is necessary to ensure that the dimensions are the same. This embodiment does not limit the numerical value of the weights. This embodiment does not limit the number of iterations and the learning rate. Training the model according to the loss function is a technical means that can be implemented by those skilled in the art, and this embodiment will not elaborate on this.

[0081] Furthermore, this embodiment provides a step for obtaining a visual composite feature matrix based on confidence scores and an LSS inverse deep network, including:

[0082] Initialize the weight vector and update each element in the initial weight vector according to the confidence score to obtain the optimal weight vector;

[0083] Based on the optimal weight vector, shallow features, mid-level features, deep features, and deep multi-scale features are fused to obtain fused multi-scale features;

[0084] A visual composite feature matrix is ​​obtained by fusing multi-scale features and an LSS inverse depth network.

[0085] Furthermore, this embodiment provides a step for updating each element in the initial weight vector based on the confidence score to obtain the optimal weight vector, including:

[0086] The current reflectivity is obtained based on a multi-source dataset.

[0087] Based on the current reflectivity and empirical parameter settings, the reflectivity correction factor is obtained;

[0088] The optimal weight vector is obtained by updating each element in the initial weight vector based on the reflection correction factor and confidence score.

[0089] Furthermore, this embodiment provides a step for updating each element in the initial weight vector based on the reflectivity correction factor and confidence score to obtain the optimal weight vector, including:

[0090] The current weight vector is obtained by updating each element in the initial weight vector based on the reflectivity correction factor and confidence score.

[0091] Based on the current weight vector, shallow features, mid-level features, deep features, and deep multi-scale features are fused to obtain the current fused multi-scale features;

[0092] The first iteration operation is performed on the current fused multi-scale features. This first iteration includes obtaining the feature response intensity and label matching success rate based on the current fused multi-scale features; obtaining the overall confidence level based on the feature response intensity and label matching success rate; determining whether the overall confidence level reaches a preset termination condition; if not, updating the current weight vector and the current fused multi-scale features according to the scaling factor, until the preset termination condition is reached, and outputting the optimal weight vector. The preset termination condition is that the overall confidence level is greater than a pre-set confidence level.

[0093] Specifically, first, the weight vector is initialized, with each weight corresponding to a feature in turn. That is, shallow features, mid-level features, deep features, and depth multi-scale features each correspond to a weight. In this embodiment, the value of the weight is not limited. Preferably, the weight can be determined according to the lighting scene category. For example, when it belongs to the case of overexposure due to supplemental lighting, the weight corresponding to the depth multi-scale feature can be increased to retain more reliable features. For example, it can be set to [0.1, 0.1, 0.2, 0.6], which correspond to shallow features, mid-level features, deep features, and depth multi-scale features, respectively.

[0094] Next, the weight vector is optimized. First, the confidence score corresponding to each feature is calculated. For example, for shallow features that focus on the edges of high diffuse visual markers, the confidence score can be obtained by statistically analyzing the proportion of edge pixels. If the dimension of the shallow feature is 240×135×64, it means that it includes 240×135 pixels, and each pixel corresponds to a 64-dimensional vector. The gradient magnitude of each vector is calculated and compared with a preset gradient magnitude. If it is greater than the preset gradient magnitude, the pixel corresponding to the vector is recorded as an edge pixel. The ratio of the number of edge pixels to the total number of pixels is recorded as the edge density. If the edge density is greater than or equal to the preset gradient magnitude, the edge density is calculated. If the density is equal to the preset density, the confidence score corresponding to the shallow feature is recorded as 1. If it is less than the preset density, the confidence score is reduced proportionally according to the difference from the preset density. In this embodiment, there are no restrictions on the preset gradient magnitude and preset density. For example, the preset gradient magnitude can be set to 0.5 and the preset density to 30%. For example, when the total number of pixels is 32,400 and the number of edge pixels is 6,480, the calculated edge density is 20%, which is lower than the preset density. The difference is 10%, and 10% is one-third of 30%. Therefore, subtracting one-third from one gives a confidence score of two-thirds, which is approximately equal to 0.67.

[0095] For mid-level features focusing on texture, the confidence score can be obtained by the variance of the local window. The smaller the variance, the more uniform the texture. For example, if the dimension of the mid-level feature is 120×68×128, it means that it includes 120×68 pixels. These pixels can be divided into multiple windows, such as each window being 4×4 in size. This embodiment does not limit this. Each pixel in each window corresponds to a 128-dimensional vector. The variance of each dimension in the vector is calculated separately. Finally, the mean of the variance of each dimension is recorded as the variance corresponding to that window. Then, the mean of the variance corresponding to each window is calculated to obtain the local variance. If the local variance is less than or equal to the preset variance, it means that the texture fluctuation is moderate and the feature is reliable. At this time, the corresponding confidence score is recorded as 1. If it is greater than the preset variance, the confidence score can be obtained by proportionally reducing the difference between the local variance and the preset variance. This embodiment does not limit the value of the preset variance. For example, it can be set to 0.2.

[0096] For semantic information and deep features of the planting rack structure focusing on high diffuse reflectance visual markers, for example, when the dimension of the deep feature is 60×34×256, each pixel corresponds to a 256-dimensional semantic vector. For each semantic vector, its similarity with the preset encoding vector is calculated, and the number of vectors with a similarity greater than the preset similarity is obtained. The ratio of the number of vectors with a similarity greater than or equal to the preset similarity is recorded as the matching success rate. If the matching success rate is greater than or equal to the preset success rate, its corresponding confidence score is recorded as 1. If it is less than the preset success rate, the confidence score is reduced proportionally according to the difference between the matching success rate and the preset success rate. This embodiment does not limit the preset success rate, for example, it is set to 90%. The preset encoding vector can be obtained from historical samples. For example, multiple high diffuse reflection visual marker samples under different lighting scene categories can be collected to make the samples cover the real scene as much as possible, ensuring the robustness of the generated vector. The deep features corresponding to each sample are obtained according to the residual network model. The data collected for each sample and the steps of obtaining deep features through the residual network model are the same as those described above, and will not be repeated in this embodiment. After obtaining each deep feature, the 256-dimensional vector of all pixels of all deep features is averaged channel by channel. The noise between samples is eliminated by averaging, and the vector obtained by averaging is finally recorded as the preset encoding vector.

[0097] For depth multi-scale features focusing on spatial contours, for example, assuming the dimensions of the depth multi-scale feature are 200×200×64, the number of effective depth pixels among all the pixels included is counted. Effective depth pixels are pixels with depth values ​​that are not 0. In the depth multi-scale feature, the depth value corresponding to a pixel is its corresponding value in the depth multi-scale feature. Finally, the ratio of the number of effective depth pixels to the total number of pixels is recorded as the depth integrity. If the depth integrity is greater than or equal to the preset integrity, the confidence score corresponding to the feature is recorded as 1. If it is less than the preset integrity, the confidence score is obtained by proportionally reducing the difference between the depth integrity and the preset integrity. In this embodiment, the value of the preset integrity is not limited, for example, it can be set to 80%.

[0098] Next, the reflectance intensity is obtained, and the ratio of the reflectance intensity to the empirical parameter tuning value is calculated. This ratio is then subtracted from 1 to obtain the reflectance correction factor. This embodiment does not limit the specific value of the empirical parameter tuning value; for example, the empirical parameter tuning value can be set to 2 to avoid the reflectance correction factor and occlusion correction factor suddenly dropping to 0. Then, each weight in the initial weight vector is adjusted based on the reflectance correction factor and the confidence score corresponding to each feature. The adjusted weights are:

[0099] ;

[0100] in, This represents the first element in the corrected initial weight vector. Each weight, This represents the first element in the initial weight vector. Each weight, Indicates the first The confidence score corresponding to each weight of the feature Indicates the reflection correction factor. The index is used to iterate through each weight in the initialized weight vector.

[0101] Then, each feature is weighted according to the corrected weights. Before weighting, the dimension of each feature needs to be unified by interpolation or convolution. Preferably, in order to further adapt to scenarios with reflections and occlusions, each feature can be enhanced to ensure computational accuracy. For example, for shallow features, edge enhancement algorithms (such as the Sobel operator) can be used to strengthen the edge details of the markers; for mid-level features, texture enhancement algorithms can be used to enhance the texture differences between the markers and crops to deal with reflections; for deep features, semantic enhancement algorithms (such as attention mechanisms) can be used to strengthen the semantic association between the planting rack structure and the markers to deal with complex scenarios; for deep multi-scale features, depth enhancement algorithms (such as bilateral filtering and dilated convolution) can be used to repair the depth loss caused by occlusion and suppress depth noise. The algorithms for enhancing features are existing technologies, and this embodiment will not elaborate on them.

[0102] like Figure 3 As shown, the weighted result is recorded as the current fused multi-scale feature. Then, the first iteration operation is performed on the current fused multi-scale feature. Taking one iteration as an example, the feature response intensity corresponding to the current fused multi-scale feature is first calculated. For example, for each pixel in the current fused multi-scale feature, there is a corresponding multi-dimensional vector. The L2 norm of each multi-dimensional vector is calculated and normalized to obtain the response value corresponding to each pixel. Then, the mean of the response values ​​corresponding to each pixel is recorded as the feature response intensity. Then, for the multi-dimensional vector corresponding to each pixel, the similarity between it and the preset encoding vector is calculated. When calculating the similarity, it is necessary to ensure that the dimensions of the multi-dimensional vector and the preset encoding vector are the same. Then, the proportion of pixels with similarity greater than the preset similarity to the total number of pixels is counted to obtain the label matching success rate. In this embodiment, the value of the preset similarity is not limited, for example, it is set to 0.9. Finally, the feature response intensity and the label matching success rate are weighted to obtain the overall confidence level corresponding to the current fused multi-scale feature. In this embodiment, the value of the weight is not limited.

[0103] Next, it is determined whether the overall confidence level is greater than the preset confidence level. If so, the iteration stops, and the current weight vector is the optimal weight vector. If not, the deep multi-scale features and shallow features are multiplied by their corresponding scaling factors to obtain the corrected weight vector. The above steps are repeated to recalculate the fused multi-scale features and the overall confidence level until the optimal weight vector is obtained. In this embodiment, the specific value of the preset confidence level is not limited. For example, it can be set to 0.7. When the overall confidence level is less than the preset confidence level, it indicates that the fusion quality is low. In this embodiment, the value of the scaling factor is not limited. For example, the scaling factor corresponding to the deep multi-scale features is 1.2, and the scaling factor corresponding to the shallow features is 0.8. Specifically, since the main reason for poor fusion quality is that shallow features are easily interfered with by reflection and crop occlusion, resulting in unreliable features, while deep multi-scale features are least affected by interference, this embodiment only adjusts these two extreme weights in order to quickly improve the fusion quality.

[0104] Then, the optimal weight vector is used for weighting to obtain the final fused multi-scale features, which are then input into the LSS (Lift, Splat, Shoot) inverse depth network to obtain the visual composite feature matrix. The LSS inverse depth network is a network specifically designed to upscale image features to the bird's-eye view (BEV) space, used to align visual features and laser BEV features in the same space. Before using the LSS inverse depth network, it needs to be trained. For example, RGBD data in multiple scenes needs to be acquired first, such as interference scenes like overlay reflection, crop occlusion, and overexposure due to supplemental lighting. At the same time, the true features of the BEV space are labeled, that is, targets such as high diffuse reflection markers, planting racks, and walls are marked in the corresponding areas of a 200×200 BEV grid as a true reference for visual BEV features. In this embodiment, the specifications of the BEV network and the network into which the point cloud data is divided are set to be consistent to facilitate subsequent fusion. Afterwards, multiple fused multi-scale features are obtained from the preprocessed RGBD data, and these features are divided into training and validation sets. The training set is then input into the LSS inverse depth network in batches. First, the Lift module outputs the inverse depth feature of each sample for each feature. The inverse depth feature includes multiple inverse depth candidate values ​​and the confidence level corresponding to each inverse depth candidate value. In this embodiment, the specific number of inverse depth candidate values ​​is not limited. For example, five can be generated. For instance, the depth range (distance from the camera to the distant planting rack) obtained by statistically analyzing the historical work of the inspection robot is usually between 1 and 50 meters. Therefore, multiple depth candidate values ​​can be generated evenly within the range of 1 to 50 meters. Then, the ratio of 1 to each depth candidate value is calculated to obtain multiple inverse depth candidate values. Next, a confidence weight is initialized for each inverse depth candidate value. The confidence weight needs to be updated during training. Therefore, in this embodiment, the value of the initialized confidence weight is not limited. For example, it can be set to 0.1. The Splat module then projects each pixel into the BEV grid based on the inverse depth candidate value and confidence weights to obtain visual BEV features. For example, the 3D world coordinates corresponding to a pixel can be obtained from the inverse depth candidate value, and then projected to obtain its corresponding position in the BEV grid. Since multiple pixels may be projected into one BEV grid cell, each BEV grid cell needs to weight the vector corresponding to each pixel in the fusion of multi-scale features according to the confidence weight. Finally, the weighted results of each BEV grid cell are integrated to obtain the visual BEV features, and the error between the visual BEV features and the true features is calculated. For example, the mean squared error can be used for calculation. Then, the mean squared error is used as the loss function, and the confidence weights are updated in reverse through a preset learning rate. This embodiment does not limit the preset learning rate. The model training is a technical step that can be implemented by those skilled in the art, and this embodiment will not elaborate on it. The Shoot module is used for dimensionality increase and does not require training.The main purpose of training the model is to enable the model to learn the confidence weights of inverse depth candidate values, solve the uncertainty of scene depth, and accurately project visual features onto the BEV grid to ensure the spatial alignment of key targets. Specifically, in inspection scenarios, reflections and crop occlusions can make it difficult to determine the actual depth of a single pixel. Through training, the network can learn to assign confidence weights to the inverse depth candidate values ​​of each pixel, thereby giving higher weights to more reliable depths, reducing the error in depth estimation under interference scenarios, and accurately projecting multi-scale features into the grid based on inverse depth features to ensure spatial alignment with subsequent fusion.

[0105] For example, in practical applications, the LSS inverse depth network first outputs trained confidence weights for each pixel's inverse depth candidate value through the Lift module, then generates visual BEV features through the Splat module, and finally the Shoot module increases the number of channels corresponding to the visual BEV features through convolution to obtain a visual composite feature matrix. In this embodiment, there is no limit to the specific number of channels. The purpose of dimensionality increase is to enable the features to carry richer and more accurate information to cope with interference. Preferably, the number of channels can be set to be the same as that of the laser composite feature matrix to avoid subsequent dimensionality changes.

[0106] This embodiment addresses the challenges of visual features being susceptible to interference from reflections and crop shading in greenhouse / inspection scenarios, and the spatial heterogeneity between visual features and laser BEV features, which prevents direct fusion. It utilizes the multi-stage structure of the ResNet-50 residual network to extract multi-scale features. Leveraging the gradient propagation advantage of residual connections, even when overexposure due to reflections distorts shallow visual features, the semantic structure of visual markers and planting racks can still be preserved through the contextual information of the deep network. Simultaneously, depth maps are stitched together to supplement spatial contour information in shading scenarios. Finally, the confidence scores of each feature are quantified for accurate evaluation. The reliability of features at different scales is estimated. Then, the correction factor is calculated by combining the reflection intensity and the weight mechanism is dynamically corrected through iterative steps, so that the weights are tilted towards more reliable features (such as multi-scale depth features) in reflective / occluded scenes, avoiding interference from invalid features. Finally, the multi-scale fused features are projected onto the BEV bird's-eye view space through the LSS inverse depth network, and the dimensionality of image features to BEV features is completed by grid weighted summation. This solves the spatial heterogeneity problem between visual features and laser BEV features, so that visual composite features still maintain high robustness and strong representation ability in reflective and occluded scenes.

[0107] Furthermore, this embodiment provides a step for obtaining a fusion matrix based on the laser composite feature matrix and the visual composite feature matrix, including:

[0108] Align the laser composite feature matrix and the visual composite feature matrix;

[0109] The current fusion matrix is ​​obtained based on the aligned laser composite feature matrix and visual composite feature matrix;

[0110] The second iteration operation is performed based on the current fusion matrix. The second iteration operation includes determining the misalignment region based on the current fusion matrix, obtaining the translation correction matrix based on the query features corresponding to each pixel in the misalignment region, updating the current fusion matrix based on the translation correction matrix, until the current fusion matrix converges, and outputting the fusion matrix.

[0111] Furthermore, this embodiment provides a step for obtaining a translation correction matrix based on the query features corresponding to each pixel in the misalignment region, including:

[0112] The weight of each pixel in the misalignment area is determined based on the binarized reflective area mask, which is obtained from a multi-source dataset.

[0113] The query features are obtained based on the weight corresponding to each pixel in the misalignment region;

[0114] Based on the query features, obtain the matching points corresponding to each pixel in the misalignment area;

[0115] Construct an error function based on the matching points;

[0116] The translation correction matrix is ​​obtained by solving the error function using the least squares method.

[0117] Specifically, the dimensions and feature distributions of the visual composite feature matrix and the laser composite feature matrix are aligned. For example, a channel mapping matrix can be set to align the dimensions of the visual composite feature matrix and the laser composite feature matrix, and the feature distribution can be aligned by moment matching. The dimension of the channel mapping matrix is ​​determined according to the number of channels corresponding to the visual composite feature matrix and the laser composite feature matrix. Moment matching is an existing technology, and this embodiment will not elaborate on it.

[0118] like Figure 4As shown, after aligning the visual composite feature matrix and the laser composite feature matrix, the second iteration operation is performed. In the first iteration, the aligned visual composite feature matrix and the laser composite feature matrix are fused to obtain the current fused matrix. Fusion means weighting the aligned visual composite feature matrix and the laser composite feature matrix according to a preset weight. Since this weight will be updated through the iteration results, this embodiment does not limit the preset weight. For example, the weight can be set to 0.5. Then, for each pixel in the two matrices, its corresponding vector is obtained, and the cross-correlation value between it and the vector of the corresponding pixel in the other matrix is ​​calculated. The closer the cross-correlation value is to 1, the more the visual and laser features of the pixel at that position match. Then, the region where the pixel with the cross-correlation value is less than the preset cross-correlation value is determined and recorded as the misalignment region. This embodiment does not limit the specific value of the preset cross-correlation value. For example, it can be set to 0.3. The calculation method of the cross-correlation value is the existing technical means, which will not be described in detail in this embodiment. Here, each pixel is equivalent to each BEV grid unit.

[0119] Then, for each pixel in the misalignment area, the corresponding query feature is calculated. For example, for one pixel, the vector corresponding to it in the aligned visual composite feature matrix and the laser composite feature matrix is ​​obtained. Then, the two vectors are weighted to obtain the corresponding query feature. This embodiment does not limit the specific value of the weight. Preferably, the weight needs to be determined based on the binarized reflective area mask. Specifically, it is first necessary to determine whether it is in the reflective area based on the binarized reflective area mask. When making the judgment, it is necessary to ensure that the binarized reflective area mask and the aligned visual composite feature matrix have the same dimension. This can be achieved through interpolation. For pixels in the reflective area, the corresponding visual feature is unreliable. Then, the vector obtained from the laser composite feature matrix should have a higher weight, such as set to 0.7. If the pixel is in the non-reflective area, it means that both the visual feature and the laser feature are reliable. The weights of the two vectors can be set to be equal.

[0120] Next, for each pixel in the misalignment region, the vector corresponding to each pixel in its neighborhood is determined in the aligned laser composite feature matrix, and the cross-correlation value between each vector and its corresponding query feature is calculated. The pixel corresponding to the maximum cross-correlation value is obtained, and this pixel is denoted as its corresponding matching point. Based on this, the pixel correspondence set of the misalignment region can be obtained. The pixel correspondence set includes the matching point corresponding to each pixel, as well as the offset between the corresponding coordinates of each pixel and its corresponding matching point. Based on the matching point and its corresponding offset, the error function between the visual features and the matching laser features is constructed according to the pixel correspondence set, and the translation correction matrix is ​​obtained by solving it using the least squares method. For example, the error function is:

[0121] ;

[0122] in, This represents the current error function. This represents the current translation correction matrix. Since this is the first iteration, the index of the translation correction matrix is ​​1. In the second iteration, the corresponding current translation correction matrix will be... , Represents the set of pixel correspondences. express The coordinates of a pixel in the image. Represents the square of the L2 norm. This represents the aligned visual composite feature matrix. Represents the coordinates in the aligned visual composite feature matrix The vector corresponding to the corresponding pixel This represents the aligned laser composite feature matrix. This represents the coordinates in the aligned laser composite feature matrix after correction by the translation correction matrix. The vector corresponding to the corresponding pixel express Translation correction value in direction, express Translation correction value in the direction.

[0123] The current translation correction matrix can then be solved using the least squares method. For example, the error function can be... about and Calculate the partial derivative and set it to 0 to obtain the translation correction for each misaligned pixel. This translation correction minimizes the loss for that pixel. Then, concatenate the translation corrections of each misaligned pixel according to their coordinate positions to obtain the current translation correction matrix. The error function corresponding to the current translation correction matrix has the minimum value. For pixels not located in the misalignment area, the values ​​of their corresponding translation corrections in the current translation correction matrix are all 0.

[0124] A translation correction matrix is ​​used to precisely align the visual features of the misaligned area with the laser features, eliminating feature offset. Then, the aligned laser composite feature matrix is ​​corrected using the translation correction matrix to obtain the final laser composite feature matrix.

[0125] Next, the aligned visual composite feature matrix is ​​fused with the final laser composite feature matrix to obtain an updated fusion matrix. For each pixel, the weights of its region are selected based on the results obtained in the first iteration, and the corresponding vectors are then fused to obtain the current fusion matrix for the second iteration. For example, when selecting weights, the cross-correlation value is recalculated and the misalignment region is determined. The steps for calculating the cross-correlation value and determining the misalignment region are the same as described above, and will not be elaborated on in this embodiment. Since the misalignment region is the region where pixels with cross-correlation values ​​less than a preset cross-correlation value are located, the region where pixels with cross-correlation values ​​greater than or equal to the preset cross-correlation value are denoted as the misalignment region. The reflective region is determined by obtaining a binarized reflective region mask. For pixels in the misaligned region, since the matching degree between the visual features and laser features of these pixels has been improved after correction, the weight corresponding to the visual features can be set higher, such as 0.7. For pixels in the aligned region, the weight corresponding to the visual features can be reduced, relying more on the laser features to maintain the stability of the aligned region and avoid introducing new offsets due to visual noise, such as setting the weight corresponding to the visual features to 0.3. For pixels in the reflective region, the weight corresponding to the visual features can be further reduced to completely avoid reflective interference, such as setting the weight corresponding to the visual features to 0.2. Since the reflective region may overlap with the aligned region or the misaligned region, reflective interference will directly destroy the information integrity of the visual features. That is, reflection will cause the edges of high diffuse reflection marks to be blurred, texture to be lost, or even overexposed to white, resulting in the loss of semantic and depth information of the visual features. Misalignment is only a positional shift between the visual and laser features. Therefore, when the reflective region may overlap with the aligned region or the misaligned region, the reflective situation should be given priority, that is, the weight corresponding to the reflective region should be used first.

[0126] After the first iteration, the iteration is terminated based on the updated fusion matrix and the current fusion matrix, i.e., whether the preset termination condition is met. If not, the aligned laser composite feature matrix and the aligned visual composite feature matrix are corrected based on the translation correction matrix, and the second iteration is performed. That is, the process of judging the misalignment region and updating the fusion matrix is ​​repeated until the current fusion matrix converges, and the final fusion matrix is ​​output. The determination of whether the preset termination condition is met, i.e., whether the fusion matrix updated in the first iteration has converged, can be done by calculating the L2 norm between the updated fusion matrix and the current fusion matrix. If the calculation result is less than the convergence threshold, the iteration stops. In this embodiment, the specific value of the convergence threshold is not limited, for example, it is set to 0.001.

[0127] This embodiment addresses the issue of low matching accuracy caused by local misalignment of cross-modal features in reflective and occluded scenarios. First, it aligns the dimensions of the visual and laser composite feature matrices using a channel mapping matrix, achieving feature distribution alignment through moment matching. This mathematically eliminates the heterogeneity barrier of cross-modal features, providing a feature foundation of the same dimension and distribution for fusion. Then, by calculating the cross-correlation value between visual and laser features, it locates local misalignment areas where the cross-correlation value is lower than a preset value, accurately identifying mismatched regions of cross-modal features. Next, it uses a binarized reflective area mask to assign weights to pixels in the misalignment areas to generate query features. Within the neighborhood of the optical feature, the matching point with the highest cross-correlation value with the query feature is found. Based on the matching corresponding point, an error function between the visual and laser features is constructed. The translation correction matrix is ​​solved by the least squares method to correct the laser features, eliminating feature offset in the misalignment area from a spatial perspective. Then, fusion weights are dynamically assigned to the misalignment area, alignment area, and reflective area. After multiple iterations and judgments on the convergence of the fusion matrix, the feature matching degree is stabilized. The final output fusion matrix retains the semantic information of the visual features and also has the geometric stability of the laser features, providing highly reliable cross-modal feature support for the pose estimation of the inspection robot.

[0128] Furthermore, this embodiment provides a method for obtaining the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step, including:

[0129] The unstable region is determined based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step.

[0130] Update the fusion matrix based on unstable regions;

[0131] The initial pose is obtained based on the updated fusion matrix;

[0132] The current position of the inspection robot is obtained based on the initial pose and the sliding window.

[0133] In this embodiment, positioning is performed every preset time interval, with a preset time interval between two adjacent moments. This embodiment does not limit the length of the preset time interval; for example, it can be set to 5 seconds. Since the robot moves slowly and the scene changes little between adjacent moments, this embodiment uses the increment between adjacent moments instead of global matching to calculate the position, thus reducing computational load. Specifically, after obtaining the final fusion matrix, the fusion matrix corresponding to the previous adjacent moment is obtained, and the similarity between the vectors at corresponding positions in the two fusion matrices is calculated. Unstable regions are determined based on the similarity. For example, regions with a similarity less than 0.3 can be considered unstable regions. 0.3 is just an example; this embodiment does not limit its specific value. The main source of unstable regions is that the reflection angle changes drastically with robot movement or lighting changes between adjacent moments, causing significant changes in the feature pattern. In this case, the error is large when calculating using the increment between adjacent moments, requiring weakened weighting. Some reflections are static and continuous, with the feature pattern remaining almost identical between adjacent moments while the robot moves. In this case, although the reflection exists, it does not disrupt the temporal stability of the features, and these features can be relied upon normally during inter-moment matching without weakened weighting.

[0134] For example, taking the weakening weighting of a pixel position in an unstable region as an example, firstly, the value corresponding to that position in the binary reflective region mask is determined, which is 1 or 0. Then, this value is multiplied by the weakening weight, and the product is subtracted by 1. Finally, the result of the subtraction is multiplied by the vector in the fusion matrix corresponding to that position at the current time to obtain the weakened weighted result for that position. In this embodiment, the specific value of the weakening weight is not limited, for example, it is set to 0.7. After obtaining the weakened weighted result for each position in the unstable region, it is used to replace the result of the corresponding position in the fusion matrix at the current time to obtain the further updated current fusion matrix. For the updated fusion matrix, the offset of each pixel between it and the adjacent fusion matrix corresponding to the previous time is calculated. Since the features obtained based on BEV correspond to the physical world, the offset of the pixel can be used to obtain the coordinate offset, and thus the initial pose can be obtained based on the coordinate offset and the position of the inspection robot at the adjacent previous time. Furthermore, since positioning by offset will result in cumulative error, it can be further corrected based on a sliding window to obtain the final pose, which is the position of the inspection robot. This embodiment does not limit the specific correction method. For example, a weighted moving average method can be used for calculation.

[0135] This embodiment considers that cross-modal fusion features between adjacent time points are prone to local instability due to reflection and occlusion. Directly locating based on feature offsets between adjacent time points would introduce unreliable interference, and incremental pose estimation has cumulative errors. Therefore, this embodiment first uses the similarity between the vectors corresponding to the fusion matrix at the current time point and the previous time point to determine unstable regions. By weakening the feature weights of these regions, the fusion matrix is ​​updated, filtering out interference from unreliable features between frames at the source. Next, relying on the one-to-one correspondence between BEV features and the physical world, the pixel offsets of the fusion matrix are directly mapped to physical coordinate offsets to quickly obtain the initial pose. This ensures the real-time performance of pose estimation and avoids errors in single-view coordinate transformation by utilizing the global consistency of the BEV space. Finally, a sliding window is used to correct the cumulative error of the incremental pose. Since the inspection robot moves slowly, the pose error within the window can cover the main cumulative error. At the same time, the sliding window only optimizes the local pose, and the computational cost is much lower than global optimization. Thus, the cumulative error is eliminated under low computational constraints, and finally, accurate, stable, and adaptable real-time pose results are obtained that are suitable for the computational power of the inspection robot.

[0136] This application provides a visual marker robust localization system for complex lighting environments, including:

[0137] The data acquisition module is used to acquire and preprocess the multi-source dataset currently corresponding to the inspection robot. The multi-source dataset includes RGBD data and point cloud data.

[0138] The calculation module is used to obtain the laser composite feature matrix based on the preprocessed point cloud data; and to obtain the visual composite feature matrix based on the preprocessed RGBD data and the residual network model.

[0139] The fusion module is used to obtain the fusion matrix based on the laser composite feature matrix and the visual composite feature matrix;

[0140] The positioning module is used to determine the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the previous time step.

[0141] This application embodiment utilizes a residual network model to extract a visual composite feature matrix, enhancing the deep semantic expression and detail representation capabilities of visual features. Simultaneously, a laser composite feature matrix constructed based on point cloud data fully preserves the geometric structure information of the environment. By fusing the laser and visual composite feature matrices, the complementary advantages of the two types of features are fully utilized, improving the robustness and comprehensiveness of environmental features. Finally, localization is achieved by matching the current and previous fused matrices, effectively solving the problem of insufficient feature representation from a single data source and significantly improving the position perception accuracy and stability of the inspection robot in complex scenarios.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A robust localization method for visual markers in complex lighting environments, characterized in that, include: The multi-source dataset corresponding to the inspection robot is acquired and preprocessed. The multi-source dataset includes RGBD data and point cloud data. The laser composite feature matrix is ​​obtained from the preprocessed point cloud data; Based on the preprocessed RGBD data and the residual network model, the visual composite feature matrix is ​​obtained; Based on the laser composite feature matrix and the visual composite feature matrix, a fusion matrix is ​​obtained; The current position of the inspection robot is obtained based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step. The step of obtaining the fusion matrix based on the laser composite feature matrix and the visual composite feature matrix includes: Align the laser composite feature matrix and the visual composite feature matrix; The current fusion matrix is ​​obtained based on the aligned laser composite feature matrix and visual composite feature matrix; The second iteration operation is performed based on the current fusion matrix. The second iteration operation includes determining the misalignment region based on the current fusion matrix, obtaining the translation correction matrix based on the query features corresponding to each pixel in the misalignment region, updating the current fusion matrix based on the translation correction matrix, until the current fusion matrix converges, and outputting the fusion matrix.

2. The robust localization method for visual markers in complex lighting environments according to claim 1, characterized in that, The step of obtaining the laser composite feature matrix based on the preprocessed point cloud data includes: The point cloud data is divided into multiple grids, and each grid includes multiple data points; The normal and curvature corresponding to each grid are obtained based on the data points; The laser composite feature matrix is ​​obtained by mapping the normals and curvatures corresponding to each grid and the fully connected layer.

3. The robust localization method for visual markers in complex lighting environments according to claim 1, characterized in that, The step of obtaining the visual composite feature matrix based on the preprocessed RGBD data and the residual network model includes: Based on the preprocessed RGBD data and the residual network model, shallow features, mid-level features, deep features, and depth multi-scale features are obtained. Calculate the confidence scores for the shallow features, mid-level features, deep features, and depth multi-scale features respectively; The visual composite feature matrix is ​​obtained based on the confidence score and the LSS inverse deep network.

4. The robust localization method for visual markers in complex lighting environments according to claim 3, characterized in that, The step of obtaining the visual composite feature matrix based on the confidence score and the LSS inverse deep network includes: Initialize the weight vector and update each element in the initial weight vector according to the confidence score to obtain the optimal weight vector; The shallow features, mid-level features, deep features, and deep multi-scale features are fused according to the optimal weight vector to obtain fused multi-scale features; Based on the fused multi-scale features and the LSS inverse depth network, a visual composite feature matrix is ​​obtained.

5. A robust visual marker localization method for complex lighting environments according to claim 4, characterized in that, The step of updating each element in the initial weight vector based on the confidence score to obtain the optimal weight vector includes: The current reflectance intensity is obtained based on the multi-source dataset; Based on the current reflectivity and empirical parameter settings, the reflectivity correction factor is obtained; The optimal weight vector is obtained by updating each element in the initial weight vector based on the reflection correction factor and the confidence score.

6. A robust visual marker localization method for complex lighting environments according to claim 5, characterized in that, The step of updating each element in the initial weight vector according to the reflectivity correction factor and the confidence score to obtain the optimal weight vector includes: The current weight vector is obtained by updating each element in the initial weight vector according to the reflection correction factor and the confidence score. Based on the current weight vector, the shallow features, mid-level features, deep features, and deep multi-scale features are fused to obtain the current fused multi-scale features; Perform a first iteration operation on the current fused multi-scale features. The first iteration operation includes obtaining the feature response intensity and label matching success rate based on the current fused multi-scale features, obtaining the overall confidence based on the feature response intensity and label matching success rate, determining whether the overall confidence reaches a preset termination condition, and if not, updating the current weight vector and the current fused multi-scale features according to the scaling factor until the preset termination condition is reached, and outputting the optimal weight vector.

7. A robust visual marker localization method for complex lighting environments according to claim 1, characterized in that, The translation correction matrix is ​​obtained based on the query features corresponding to each pixel in the misalignment region, including: The weight corresponding to each pixel in the misalignment area is determined based on the binarized reflective area mask, wherein the binarized reflective area mask is obtained based on the multi-source dataset; The query feature is obtained based on the weight corresponding to each pixel in the misalignment region; Based on the query features, obtain the matching corresponding point for each pixel in the misalignment region; Construct an error function based on the matching points; The translation correction matrix is ​​obtained by solving the error function using the least squares method.

8. A robust visual marker localization method for complex lighting environments according to claim 1, characterized in that, The step of obtaining the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step includes: The unstable region is determined based on the fusion matrix and the fusion matrix corresponding to the adjacent previous time step. The fusion matrix is ​​updated based on the unstable region; The initial pose is obtained based on the updated fusion matrix; The current position of the inspection robot is obtained based on the initial pose and the sliding window.

9. A robust visual marker localization system for complex lighting environments, used to implement the robust visual marker localization method for complex lighting environments as described in any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire and preprocess the multi-source dataset currently corresponding to the inspection robot. The multi-source dataset includes RGBD data and point cloud data. The calculation module is used to obtain the laser composite feature matrix based on the preprocessed point cloud data; Based on the preprocessed RGBD data and the residual network model, the visual composite feature matrix is ​​obtained; The fusion module is used to obtain a fusion matrix based on the laser composite feature matrix and the visual composite feature matrix; The positioning module is used to obtain the current position of the inspection robot based on the fusion matrix and the fusion matrix corresponding to the previous adjacent time step.

Images