An Unmanned Mining Truck Positioning Method, Device, Equipment and Medium

Abstract

The present invention belongs to the technical field of artificial intelligence, and specifically relates to a positioning method, device, equipment and medium for driverless mining trucks. The positioning method for driverless mining trucks of the present invention first obtains multiple front road images and infrared images within a preset distance range of the front road environment during the driving of the mining truck; after obtaining the front road images, the deep learning algorithm SuperPoint is used to extract the feature key points of each image, and then SuperGlue is used to match the key points of adjacent images, so as to obtain the conditions for pose estimation of the on-vehicle camera. The infrared sensing is used to correct the shadow misjudgment of the real-time map, and finally the real-time position change of the mining truck is obtained, which can correct the GNSS positioning and be used as an alternative positioning means in places with poor signals in mountainous areas.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to an unmanned mining truck positioning method, device, equipment and medium. Background Art

[0002] The unmanned driving of coal mine trucks means that unmanned mining trucks transport the sandstone above the coal seam to the waste dump for disposal, and after the raw coal is exposed, use the mining trucks to transport it to the crushing station for crushing and separation. In this process, the non-integrity constraints of the vehicle are followed to plan an optimal or sub-optimal path without collision from the starting position state to the target position state. Environmental perception, positioning and navigation, and path planning are several key technologies for realizing unmanned mining trucks. However, the mining area often has problems such as complex environment, poor road conditions, and coal ash covering the roads, making it difficult to distinguish shadows and roads. Therefore, unmanned mining trucks need to have good environmental perception capabilities. With the change of the mining progress, the roads and working surfaces where the unmanned mining trucks are located are constantly changing. At the same time, there may be falling gravel, resulting in a certain area being blocked and no longer passable, or the GNSS cannot be used due to weak signals in mountainous areas. Therefore, it is necessary to use positioning and navigation for real-time map updating and path planning that can obtain better strategies.

[0003] Common perceptions of unmanned mining trucks include laser, ultrasonic, infrared, visual image and other methods. For different environments, these methods have their own advantages and disadvantages: the visual images obtained by visible light cameras are suitable for short-distance measurement, but are greatly affected by bad weather and dust. The advantage is that the target range and contour can be accurately extracted, and obstacles can be better identified and a real-time 3D map can be constructed in cooperation with image algorithms; lidar has high resolution and strong anti-interference ability, but the beam is narrow and it is easy to lose information such as the target contour; ultrasonic radars have relatively low costs, but weak resolution ability, low detection accuracy, and too short ranging; infrared sensors can well cope with night and bad weather conditions, but have low ranging and speed measurement accuracy. The perception results are used as the analysis benchmark for the positioning and navigation of unmanned mining trucks.

[0004] Common positioning and navigation mainly include inertial navigation positioning, GNSS differential positioning, and simultaneous localization and mapping technology (SLAM). Inertial navigation positioning relies on relatively expensive IMU components, and the error becomes larger as it is used for a longer time. GNSS performs satellite positioning, has the characteristics of all-round and all-weather, and strong anti-interference ability. Currently, the problem is that the positioning accuracy is not yet fully competent. SLAM technology mainly includes laser SLAM and visual SLAM, which are also the current mainstream positioning technologies. They can fuse multiple sensors, have strong environmental adaptability, high positioning accuracy, and broad application prospects.

[0005] Currently, the commonly used visual SLAM algorithms are difficult to be applied to situations where the road conditions are relatively single (such as the mining area roads are basically covered with coal ash) and the lighting conditions change greatly (non-urban areas or indoors). Summary of the Invention

[0006] The present invention provides a positioning method, device, equipment and medium for driverless mining trucks, equipped with an infrared detection system as an auxiliary to GNSS positioning, so as to solve the problem of inaccurate positioning of driverless mining trucks in weak signal areas.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a positioning method for driverless mining trucks, including the following steps:

[0009] Obtain the front road image and infrared image during the driving process of the driverless mining truck;

[0010] Use the pre-trained SuperPoint deep neural network to extract feature points in the front road image;

[0011] According to the positions and descriptor information of the feature points in the front road image, use the preset feature matching network SuperGlue to match the feature points in two adjacent front road images to obtain matching feature points;

[0012] Convert the front road image and the infrared image into grayscale images respectively, compare the grayscale images of the front road image and the infrared image, identify the positions where the difference exceeds the preset standard as shadow parts, and delete the matching feature points on the shadow parts;

[0013] Based on the remaining matching feature points after removing the shadow parts, determine the movement trajectory of the driverless mining truck.

[0014] Further, in the step of using the pre-trained SuperPoint deep neural network to extract feature points in the front road image, the training method of the SuperPoint deep neural network is as follows:

[0015] Use the MagicPoint network to pre-construct an artificial synthetic image sample library; import the artificial synthetic image samples in the artificial synthetic image sample library into the untrained SuperPoint deep neural network for preliminary training;

[0016] Obtain multiple pre-shot target road images, mark feature points on each target road image, use homography transformation on the target road image to make the target road image present different angles; use the MagicPoint network to train the target road image and the marked feature points to obtain the detector of the MagicPoint network, and use the detector of the MagicPoint network as the detector of the SuperPoint deep neural network;

[0017] Among them, the SuperPoint deep neural network includes an encoder and a decoder; the encoder is used to reduce the image space scale of the target road image while increasing the number of channels; the decoder is divided into two parts. The first decoder outputs the probability that each pixel point in the target road image is a feature point; the second decoder uses the bilinear interpolation algorithm for the feature points to obtain the descriptors of the feature points, and uses the L2 norm to unify the lengths of the descriptors;

[0018] Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, and optimize the SuperPoint deep neural network based on the optimization direction. After the optimization is completed, the trained SuperPoint deep neural network is obtained.

[0019] Furthermore, the result error obtained by the two decoders is calculated according to the following formula:

[0020]

[0021] Among them, is the error for judging whether it is a feature point; is the descriptor error; λ is used to weight the observation of key feature points and the description of feature points; X is the true value of the feature points extracted from the target road image; X′ is the true value of the feature points in the target road image after homography transformation; D is the value of the descriptor extracted from the target road image; Y is the true value of the feature points extracted from the adjacent target road image; Y′ is the true value of the feature points in the adjacent target road image after homography transformation; S is the correlation between two adjacent target road images.

[0022] Furthermore, the step of using the preset feature matching network SuperGlue to match the feature points in two adjacent front road images specifically includes:

[0023] Use the SuperPoint deep neural network to detect the feature point positions p and descriptors d of two adjacent front road images, and generate correlation vectors based on the feature point positions p and descriptors d;

[0024] Use the multiplex graph neural network to connect the edges of two nodes on the same front road image as internal edges, and the set is denoted as ε self ; Use the multiplex graph neural network to connect the edges of feature points in different images, and the set is denoted as ε cross ; Based on the union ε self and ε crosS generate the aggregation result m of all feature points on ε ε→i; wherein, the nodes of the multiplexed graph neural network are jointly composed of the feature points of two front road images;

[0025] Based on the aggregation result m ε→i Update the feature matching vectors of each feature point and its corresponding descriptor;

[0026] Based on the feature matching vectors of the front road image A and the front road image B, use a preset constraint relationship to obtain a matching feature point assignment matrix; wherein, the front road image A and the front road image B are adjacent.

[0027] Further, the step of determining the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow part specifically includes:

[0028] Based on the coordinates of the matching feature points, use the five-point method to calculate the pose transformation of the camera; wherein, the pose transformation includes the rotation angle Ω and the displacement distance t of the camera, and is represented by the essential matrix E;

[0029] After obtaining the essential matrix E, use the method of singular value decomposition to calculate the rotation angle Ω and the displacement distance t to obtain the movement trajectory of the vehicle.

[0030] Further, after the step of determining the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow part, the following steps are further included:

[0031] Use a Kalman filter to fuse the GNSS positioning result and the movement trajectory of the driverless mining truck to obtain a fusion result;

[0032] Determine the position positioning of the driverless mining truck according to the fusion result.

[0033] In a second aspect, the present invention provides a driverless mining truck positioning device, including:

[0034] An image acquisition module for acquiring the front road image and the infrared image during the driving of the driverless mining truck;

[0035] A feature extraction module for using a pre-trained SuperPoint deep neural network to extract feature points in the front road image;

[0036] A feature matching module for matching the feature points in two adjacent front road images according to the positions and descriptor information of the feature points on the front road image, using a preset feature matching network SuperGlue to obtain matching feature points;

[0037] A comparison module, configured to convert the forward road image and the infrared image into grayscale images respectively, compare the grayscale image of the forward road image with the grayscale image of the infrared image, recognize the positions where the difference exceeds a preset standard as shadow parts, and delete the matching feature points on the shadow parts;

[0038] A positioning module, configured to determine the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow parts.

[0039] Further, in the feature extraction module, the training method of the SuperPoint deep neural network is as follows:

[0040] Use the MagicPoint network to pre-construct an artificial synthetic image sample library; import the artificial synthetic image samples in the artificial synthetic image sample library into the untrained SuperPoint deep neural network for preliminary training;

[0041] Obtain multiple pre-shot target road images, mark feature points on each of the target road images, use a homography transformation on the target road images to make the target road images present different angles; use the MagicPoint network to train for the target road images and the marked feature points to obtain a detection sub-module of the MagicPoint network, and use the detection sub-module of the MagicPoint network as the detection sub-module of the SuperPoint deep neural network;

[0042] Among them, the SuperPoint deep neural network includes an encoder and a decoder; the encoder is used to reduce the image space scale of the target road image while increasing the number of channels; the decoder is divided into two parts, the first decoder outputs the probability that each pixel point in the target road image is a feature point; the second decoder uses a bilinear interpolation algorithm for the feature points to obtain the descriptor of the feature points, and uses the L2 norm to unify the length of the descriptors;

[0043] Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, optimize the SuperPoint deep neural network based on the optimization direction, and after the optimization is completed, obtain the trained SuperPoint deep neural network.

[0044] In a third aspect, the present invention provides an electronic device, including a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the above-mentioned driverless mining truck positioning method.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium, and the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the above-mentioned driverless mining truck positioning method is implemented.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] The unmanned mining truck positioning method provided by the present invention first obtains multiple front road images and infrared images within a preset distance range of the front road environment during the driving of the mining truck; after obtaining the front road images, the deep learning algorithm SuperPoint is used to extract the feature key points of each picture, and then SuperGlue is used to match the key points of adjacent pictures, so as to obtain the conditions for pose estimation of the on-vehicle camera. The infrared perception is used to correct the shadow misjudgment of the real-time map, and finally the real-time position change of the mining truck is obtained, which can correct the GNSS positioning and be used as an alternative positioning means in places with poor signals in mountainous areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The specification drawings forming a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0049] Figure 1 is a flow example diagram of an unmanned mining truck positioning method according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the SuperPoint deep neural network model in an embodiment of the present invention;

[0051] Figure 3 is a structural block diagram of an unmanned mining truck positioning device according to an embodiment of the present invention;

[0052] Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments. It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0054] The following detailed descriptions are all exemplary descriptions, aiming to provide further detailed descriptions of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0055] Embodiment 1

[0056] As Figure 1 shown, the present invention provides an unmanned mining truck positioning method, including the following steps:

[0057] Step S1: Obtain the front road image and infrared image during the driving of the driverless mining truck.

[0058] Specifically, multiple consecutive front road images within a preset distance range in the front road environment of the driverless mining truck can be obtained through an on-vehicle monocular camera; multiple consecutive infrared images within a preset distance range in the front road environment can be obtained through an on-vehicle infrared sensor or infrared camera.

[0059] Step S2: Use the pre-trained SuperPoint deep neural network to extract feature points from the front road image.

[0060] Specifically, the training method of the SuperPoint deep neural network in this solution is as follows:

[0061] 1) Use the MagicPoint network to pre-construct an artificial synthetic image sample library; among them, the artificial synthetic image sample library includes artificial synthetic image samples, such as some different shapes, figures, etc. Specifically, they can be shapes such as triangles, circles, pyramids, etc.; import the artificial synthetic image samples into the untrained SuperPoint deep neural network for preliminary training, so that the SuperPoint deep neural network can perform some basic feature point detections; in this solution, the MagicPoint network is a fully convolutional neural network.

[0062] 2) Obtain multiple pre-shot target road images, mark some feature points on each target road image respectively, use homography transformation on the target road images with marked feature points to make these target road images present different angles; then use the MagicPoint network to train these target road images and the marked feature points again. The entire step 2) will be repeated multiple times, and finally, the detection sub of the MagicPoint network is obtained, and the detection sub of the MagicPoint network is used as the detection sub of the SuperPoint deep neural network.

[0063] 3) The above-mentioned SuperPoint deep neural network includes an encoder and a decoder;

[0064] The encoder is used to change the image space size of the target road image from H*W to Hc*Wc, so that the image space scale is reduced while the number of channels increases; specifically, H / 8 = Hc and W / 8 = Wc can be taken; H is the height and W is the width.

[0065] The decoder is divided into two parts:

[0066] The first decoder is used to output the probability that each pixel point of the target road image is a feature point. Specifically, through two convolutions, the target road image finally has 65 channel lengths (with widths and heights of Wc and Hc respectively), obtaining non-overlapping local 8*8 pixel grid regions, plus 1 channel corresponding to the non-detection of feature points in the 8*8 region. Then, the 1 channel without feature points is removed (at this time, the number of channels is 64). The Softmax function is used to map the score of each pixel point to the range [0,1], obtaining the probability that each pixel point is a feature point. Each pixel point of all 64 maps is taken out and assembled into an 8*8 map, and they are merged together so that the final size of the image is H*W.

[0067] The second decoder is used to obtain the descriptor of the feature point. Specifically, through two convolutions, the structure of the final image is Wc*Hc*256. The bilinear interpolation algorithm is used for the feature point to obtain the descriptor, and the L2 norm is used to unify the length of the descriptor.

[0068] 4) Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, and optimize the SuperPoint deep neural network based on the optimization direction. After the optimization is completed, the trained SuperPoint deep neural network is obtained.

[0069] Specifically, the result error obtained by the two decoders is calculated according to the following formula:

[0070]

[0071] Among them, is the error for judging whether it is a feature point, and it is a cross-entropy loss function; is the descriptor error, which is obtained by comparing the results obtained from two target road images after homography transformation with the actual descriptor. λ is used to weight the optimization of the observation ability for key feature points and the description ability for feature points; X is the true value of the feature points extracted from the target road image; X′ is the true value of the feature points in the target road image after homography transformation; D is the value of the descriptor extracted from the target road image; Y is the true value of the feature points extracted from the adjacent target road image; Y′ is the true value of the feature points in the adjacent target road image after homography transformation; S is the correlation between two adjacent target road images.

[0072] Step S3: According to the position and descriptor information of the feature points on the front road image, use the preset feature matching network SuperGlue to match the feature points in two adjacent front road images, obtaining the matching feature points, and the matching feature points are used for vehicle position estimation.

[0073] Specifically, the feature points p and descriptors d of two adjacent front road images are detected by the SuperPoint deep neural network, and the feature matching vector x of a single feature point is generated based on the feature points p and descriptors d;

[0074] x = d + MLP(p)

[0075] where MLP is a multi-layer perception encoder that can elevate the feature points p to high-dimensional vectors and then add them to the descriptors d.

[0076] The edges connecting two nodes on the same front road image using the multiplex graph neural network contained in the SuperPoint deep neural network are internal edges, and the set is denoted as ε self ; The edges connecting the feature points of different images using the multiplex graph neural network, and the set is denoted as ε cross ; Based on ε self and the union ε of ε cross , the aggregation result m of all feature points on ε is generated ε→i ; where the nodes of the multiplex graph neural network are jointly composed of the feature points of two front road images;

[0077] Based on the aggregation result m ε→i Update the feature matching vector of each feature point and its corresponding descriptor as the representation vector, as shown in the following formula:

[0078]

[0079] where represents the manifestation form of feature i in front road image A at the l-th layer; [·||·] represents the concatenation operation. Starting from l = 1 layer, when l is odd, ε = ε self , when l is even, ε = ε cross ; Simulate the process of humans browsing and searching for features, and alternately aggregate and update along and between front road images.

[0080] For example, after the feature i of front road image A passes through the multiplex graph neural network, the feature matching vector f is obtained according to the following formula i A :

[0081]

[0082] where W and b are the weight and bias respectively;

[0083] According to the same method, the feature matching vector of front road image B is obtained as

[0084] Based on the feature matching vectors of the front road image A and the front road image B, a matching feature point allocation matrix is obtained by using a preset constraint relationship; wherein, the front road image A and the front road image B are adjacent.

[0085] The constraints include: at most, a feature point in one front road image corresponds to one feature point in the other front road image; when some feature points cannot find corresponding matching feature points due to objective factors, they are ignored; the Sinkhorn algorithm is used with f i A and to obtain the corresponding feature point allocation matrix of the feature points that can find corresponding matches.

[0086] Step S4: Convert the front road image and the infrared image into grayscale images respectively, compare the grayscale images of the front road image and the infrared image, identify the positions where the difference exceeds the preset standard as shadow parts, and delete the matching feature points on the shadow parts.

[0087] Specifically, when comparing the grayscale images of the front road image and the infrared image, a standard for the degree of difference is preset based on manual experience, and the positions exceeding the standard are identified as shadow parts. There may be some matching feature points on the shadow parts, and these matching feature points are deleted.

[0088] Step S5: Determine the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow parts.

[0089] Specifically, based on the coordinates of 5 or more matching feature points, the pose transformation of the camera is calculated using the five-point method; as an example, the coordinate form of a pair of matching feature points is (x i , y i )(x′ i ,, y′ i ).

[0090] Among them, the pose transformation includes the rotation angle Ω and the displacement distance t of the camera, which is represented by the essential matrix E; after obtaining the essential matrix E, the rotation angle Ω and the displacement distance t are calculated using the method of singular value decomposition to obtain the movement trajectory of the vehicle.

[0091] Specifically, according to the epipolar constraint, the relationship can be obtained:

[0092]

[0093] where contains the coordinate information of 5 corresponding matching feature points, and the essential matrix E is a 3×3 matrix.

[0094] After obtaining the intrinsic matrix E, the rotation angle Ω and displacement distance t are calculated using the singular value decomposition method as follows:

[0095] E = ULV T

[0096] t = ULWV T

[0097] Ω = ULW -1 V T

[0098] Among them, U is an m×m unitary matrix; V T is the conjugate transpose of V, which is an n×n unitary matrix; L is a semi - positive definite m×n diagonal matrix. The elements Li on the diagonal of L are the singular values of E; W is a square matrix used to adjust the position of the internal information of the matrix:

[0099]

[0100] In an alternative embodiment of the present invention, a Kalman filter is used to fuse the GNSS positioning result and the movement trajectory of the unmanned mining truck to obtain a fusion result; the position of the unmanned mining truck is determined according to the fusion result. If the signal is weak in mountainous areas and GNSS cannot be used, the result of step 106 is directly used as the vehicle's position positioning.

[0101] The positioning method of the unmanned mining truck in this solution processes the results of the vehicle - mounted camera using the SuperPoint and SuperGlue deep learning algorithms to obtain visual SLAM and correct the shadow misjudgment through infrared sensing. Finally, it can correct the GNSS positioning and be used as an alternative positioning means in areas with poor mountain signal.

[0102] Embodiment 2

[0103] As Figure 3 shown, based on the same inventive concept as the above - mentioned embodiment, this solution also provides an unmanned mining truck positioning device, including:

[0104] An image acquisition module for acquiring the front - road image and infrared image during the driving of the unmanned mining truck.

[0105] A feature extraction module for using the pre - trained SuperPoint deep neural network to extract feature points in the front - road image.

[0106] In the feature extraction module, the training method of the SuperPoint deep neural network is as follows:

[0107] Pre - construct a synthetic image sample library using the MagicPoint network; import the synthetic image samples in the synthetic image sample library into an untrained SuperPoint deep neural network for preliminary training;

[0108] Obtain pre - captured target road images, mark feature points on multiple target road images, perform a homography transformation on the target road images to make the target road images present different angles; use the MagicPoint network to train the target road images and the marked feature points to obtain the detector of the MagicPoint network, and use the detector of the MagicPoint network as the detector of the SuperPoint deep neural network;

[0109] Among them, the SuperPoint deep neural network includes an encoder and a decoder; the encoder is used to reduce the image space scale of the target road image while increasing the number of channels; the decoder is divided into two parts. The first decoder outputs the probability that each pixel point in the target road image is a feature point; the second decoder uses a bilinear interpolation algorithm for the feature points to obtain the descriptors of the feature points, and uses the L2 norm to unify the lengths of the descriptors;

[0110] Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, optimize the SuperPoint deep neural network based on the optimization direction, and after the optimization is completed, obtain the trained SuperPoint deep neural network.

[0111] A feature matching module, which is used to match the feature points in two adjacent front - road images according to the positions and descriptor information of the feature points on the front - road images, using a preset feature matching network SuperGlue to obtain matching feature points;

[0112] A comparison module, which is used to convert the front - road image and the infrared image into grayscale images respectively, compare the grayscale image of the front - road image and the grayscale image of the infrared image, recognize the positions where the difference exceeds a preset standard as shadow parts, and delete the matching feature points on the shadow parts;

[0113] A positioning module, which is used to determine the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow parts.

[0114] Embodiment 3

[0115] As Figure 4As shown in the figure, the present invention further provides an electronic device 100 for implementing an unmanned mining truck positioning method; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104. The memory 101 can be used to store the computer program 103. The processor 102 realizes the steps of an unmanned mining truck positioning method in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0116] The memory 101 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device 100 (such as audio data, etc.). In addition, the memory 101 may include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0117] At least one processor 102 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or the processor 102 may also be any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and connects various parts of the entire electronic device 100 through various interfaces and lines.

[0118] The memory 101 in the electronic device 100 stores multiple instructions to implement an unmanned mining truck positioning method, and the processor 102 can execute the multiple instructions to achieve:

[0119] Obtain multiple front road images and infrared images within a preset distance range of the front road environment during the driving of the mining truck;

[0120] Use the pre-trained SuperPoint deep neural network to extract feature points on the forward road image of the unmanned mining truck's forward road;

[0121] According to the positions and descriptor information of the feature points on the forward road of the unmanned mining truck, use the pre-trained feature matching network SuperGlue to match the feature points in two adjacent forward road images to obtain matching feature points;

[0122] Convert the forward road image and the infrared image into grayscale images respectively, compare the grayscale images of the forward road image and the infrared image, identify the positions where the difference exceeds the preset standard as shadow parts, and delete the matching feature points on the shadow parts;

[0123] Based on the remaining matching feature points after removing the shadow parts, determine the movement trajectory of the unmanned mining truck to obtain the position positioning of the unmanned mining truck.

[0124] Embodiment 4

[0125] If the modules / units integrated in the electronic device 100 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory and read-only memory (ROM, Read-Only Memory).

[0126] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0128] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows and / or blocks Figure 1 in one or more flows and / or blocks Figure 1 or means for implementing the functions specified in one or more blocks.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent replacements can still be made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A positioning method for driverless mining trucks, characterized in that, It includes the following steps: Obtain the front road image and infrared image during the driving of the driverless mining truck; Use the pre-trained SuperPoint deep neural network to extract feature points in the front road image; According to the positions and descriptor information of the feature points in the front road image, use the preset feature matching network SuperGlue to match the feature points in two adjacent front road images to obtain matching feature points; Convert the front road image and the infrared image into grayscale images respectively, compare the grayscale images of the front road image and the infrared image, identify the positions where the difference exceeds the preset standard as shadow parts, and delete the matching feature points on the shadow parts; Based on the remaining matching feature points after removing the shadow parts, determine the moving trajectory of the driverless mining truck.

2. The unmanned mining truck positioning method according to claim 1, wherein In the step of using the pre-trained SuperPoint deep neural network to extract feature points in the front road image, the training method of the SuperPoint deep neural network is as follows: Use the MagicPoint network to pre-construct an artificial synthetic image sample library; import the artificial synthetic image samples in the artificial synthetic image sample library into the untrained SuperPoint deep neural network for preliminary training; Obtain multiple pre-shot target road images, mark feature points on each of the target road images, perform homography transformation on the target road images to make the target road images present different angles; use the MagicPoint network to train the target road images and the marked feature points to obtain the detector of the MagicPoint network, and use the detector of the MagicPoint network as the detector of the SuperPoint deep neural network; Among them, the SuperPoint deep neural network includes an encoder and a decoder; the encoder is used to reduce the image space scale of the target road image and increase the number of channels; the decoder is divided into two parts, the first decoder outputs the probability that each pixel point in the target road image is a feature point; the second decoder uses the bilinear interpolation algorithm for the feature points to obtain the descriptors of the feature points, and uses the L2 norm to make the descriptors of the same length; Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, optimize the SuperPoint deep neural network based on the optimization direction, and after the optimization is completed, obtain the trained SuperPoint deep neural network.

3. The unmanned mining truck positioning method according to claim 2, wherein The result error obtained by the two decoders is calculated according to the following formula: wherein, is the error for determining whether it is a feature point; is the descriptor error; is used for weight optimization of the observation ability for key feature points and the description ability for feature points; X is the true value of the feature points extracted from the target road image; is the true value of the feature points in the target road image after homography transformation; D is the value of the descriptor extracted from the target road image; Y is the true value of the feature points extracted from the adjacent target road image; is the true value of the feature points in the adjacent target road image after homography transformation; S is the correlation between two adjacent target road images.

4. The positioning method of the unmanned mining truck according to claim 1, wherein, The step of using the preset feature matching network SuperGlue to match the feature points in two adjacent front road images specifically includes: Use the SuperPoint deep neural network to detect the feature point positions p and descriptors d of two adjacent front road images, and generate correlation vectors based on the feature point positions p and descriptors d; The edge connecting two nodes on the same front road image using a multiplexed graph neural network is an inner edge, and the set is denoted as ; Use multiplexed graph neural networks to connect the edges of feature points of different images, and the set is recorded as Based on the and The union of Generate all feature points in Aggregation results on ; Wherein, the nodes of the multiplexed graph neural network are composed of feature points of two images of the road ahead; Based on the aggregation result Update the feature matching vectors of each feature point and its corresponding descriptor; Based on the feature matching vectors of the front road image A and the front road image B, a matching feature point allocation matrix is obtained by using a preset constraint relationship; wherein, the front road image A and the front road image B are adjacent.

5. The positioning method of the driverless mining truck according to claim 1, wherein The step of determining the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow part specifically includes: Based on the coordinates of the matched feature points, calculate the pose transformation of the camera using the five-point method; wherein, the pose transformation includes the rotation angle and the displacement distance t, which is represented by the essential matrix E; After obtaining the intrinsic matrix E, the rotation angle is calculated using the singular value decomposition method. and the displacement distance t to obtain the vehicle's movement trajectory.

6. The positioning method of the driverless mining truck according to claim 1, wherein After the step of determining the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow part, the following steps are further included: Using a Kalman filter to fuse the GNSS positioning result and the movement trajectory of the driverless mining truck to obtain a fusion result; Determining the position positioning of the driverless mining truck according to the fusion result.

7. An unmanned mining truck positioning device, characterized in that, Including: An image acquisition module for acquiring the front road image and the infrared image during the driving of the driverless mining truck; A feature extraction module for using the pre-trained SuperPoint deep neural network to extract the feature points in the front road image; A feature matching module for matching the feature points in two adjacent front road images using a preset feature matching network SuperGlue according to the positions and descriptor information of the feature points on the front road image to obtain matching feature points; A comparison module for respectively converting the front road image and the infrared image into grayscale images, comparing the grayscale image of the front road image and the grayscale image of the infrared image, identifying the positions where the difference exceeds the preset standard as the shadow part, and deleting the matching feature points on the shadow part; A positioning module for determining the movement trajectory of the driverless mining truck based on the remaining matching feature points after removing the shadow part.

8. The unmanned mining truck positioning device according to claim 7, characterized in that, In the feature extraction module, the training method of the SuperPoint deep neural network is as follows: Using the MagicPoint network to pre-construct an artificial synthetic image sample library; importing the artificial synthetic image samples in the artificial synthetic image sample library into the untrained SuperPoint deep neural network for preliminary training; Obtaining multiple pre-shot target road images, marking the feature points on each of the target road images, using a homography transformation on the target road images to make the target road images present different angles; using the MagicPoint network to train the target road images and the marked feature points to obtain the detector of the MagicPoint network, and using the detector of the MagicPoint network as the detector of the SuperPoint deep neural network; Wherein, the SuperPoint deep neural network includes an encoder and a decoder; the encoder is used to reduce the image space scale of the target road image and increase the number of channels; the decoder is divided into two parts, the first decoder outputs the probability that each pixel point in the target road image is a feature point; the second decoder uses a bilinear interpolation algorithm for the feature points to obtain the descriptors of the feature points, and uses the L2 norm to unify the lengths of the descriptors; Determine the optimization direction of the SuperPoint deep neural network according to the result error obtained by the two decoders, optimize the SuperPoint deep neural network based on the optimization direction, and after the optimization is completed, obtain the trained SuperPoint deep neural network.

9. An electronic device, characterized in that, It includes a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the unmanned mining truck positioning method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, it implements the unmanned mining truck positioning method according to any one of claims 1 to 6.

Images