Downhole vehicle positioning method, apparatus, device, and storage medium

Abstract

The present disclosure provides a downhole vehicle positioning method, device, equipment and storage medium, and relates to the technical field of coal mining. In some embodiments of the present disclosure, vehicle electronic control unit information, vehicle video stream information and vehicle ultra-wideband positioning information of a target vehicle are obtained; initial inertial positioning position information of the target vehicle is determined according to the vehicle electronic control unit information; initial visual positioning position information of the target vehicle is determined according to the vehicle video stream information; the initial inertial positioning position information is globally coordinate-converted to obtain target inertial positioning position information, and the initial visual positioning position information is globally coordinate-converted to obtain target visual positioning position information; the target inertial positioning position information, the target visual positioning position information and the vehicle ultra-wideband positioning information are fused to obtain vehicle positioning fusion position information of the target vehicle; and the present disclosure improves the positioning accuracy of the vehicle in a motion state.

Description

Technical Field

[0001] This disclosure relates to the field of coal mining technology, and in particular to a method, apparatus, equipment and storage medium for positioning underground vehicles. Background Technology

[0002] With the continuous advancement of intelligent construction in coal mines, the automation and intelligence levels of underground transportation systems are constantly improving. Precise positioning of underground transport vehicles such as trackless rubber-tired vehicles and rail-mounted electric locomotives has become a key technology for ensuring safe production and improving transportation efficiency. Due to the complex underground environment of coal mines, characterized by narrow roadways, poor lighting conditions, strong electromagnetic interference, and lack of GPS signals, traditional wireless signal-based positioning methods are insufficient to meet the high-precision and high-stability positioning requirements of underground vehicles during dynamic operation. Therefore, integrating multiple positioning methods, especially introducing AI vision-based positioning technology, has become an important research direction for improving the positioning capabilities of underground vehicles.

[0003] Currently, vehicle positioning in underground coal mines primarily relies on wireless positioning technologies such as inertial navigation systems (INS), radio frequency identification (RFID), Wi-Fi fingerprint positioning, and ultra-wideband (UWB). Among these, UWB is widely used in underground positioning due to its high ranging accuracy; some systems employ a fusion of UWB and INS, using algorithms such as Kalman filtering to improve positioning stability. In addition, some mines are experimenting with visual-assisted positioning methods, utilizing fixed markers on tunnel walls or track features to estimate vehicle positions using traditional image processing methods.

[0004] Currently, the positioning accuracy of underground vehicles in coal mines is relatively low when the vehicles are in motion. Summary of the Invention

[0005] This disclosure provides a method, apparatus, equipment, and storage medium for locating underground vehicles, in order to at least solve the problem of low positioning accuracy of existing underground vehicles.

[0006] The technical solution disclosed herein is as follows:

[0007] This disclosure provides a method for locating underground vehicles, including:

[0008] Acquire information about the target vehicle's electronic control unit, in-vehicle video stream, and ultra-wideband positioning.

[0009] Based on the vehicle electronic control unit information, the initial inertial positioning position information of the target vehicle is determined;

[0010] Based on the in-vehicle video stream information, the initial visual positioning location information of the target vehicle is determined;

[0011] The initial inertial positioning information is transformed into global coordinates to obtain the target inertial positioning information, and the initial visual positioning information is transformed into global coordinates to obtain the target visual positioning information.

[0012] The target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion information of the target vehicle.

[0013] Optionally, the vehicle electronic control unit information includes: specific force and angular velocity; determining the initial inertial positioning information of the target vehicle based on the vehicle electronic control unit information includes:

[0014] Obtain the given initial conditions;

[0015] The given initial conditions, the specific force, and the angular velocity are input into the trajectory recursion algorithm to obtain the initial inertial positioning information of the target vehicle.

[0016] Optionally, determining the initial visual positioning information of the target vehicle based on the in-vehicle video stream information includes:

[0017] Based on the vehicle video stream information, sign detection and sign tracking are performed to obtain the target sign and the relative position of the target sign and the target vehicle.

[0018] The target marker is identified to determine its type;

[0019] The initial visual positioning information of the target vehicle is determined based on the type of the target marker and its relative position.

[0020] Optionally, the step of detecting and tracking landmarks based on the vehicle video stream information to obtain the target landmark and its relative position to the target vehicle includes:

[0021] The in-vehicle video stream information is subjected to data augmentation processing, and a set of anchor presets is initialized to obtain an input image for target detection;

[0022] The input image is used to extract features using a feature extraction network to obtain a multi-scale feature map.

[0023] The multi-scale feature maps are fused using the PANet structure to obtain fused feature map information;

[0024] Based on the classification loss function and the bounding box regression loss function, the parameters of the target detection network are adjusted by an optimizer to predict the category and location information of the target marker.

[0025] Based on the detection results of the current frame and historical frames, the motion state of the target marker is predicted using Kalman filtering;

[0026] Based on the intersection-union distance between the predicted bounding box and the detection bounding box in the current frame, the Hungarian algorithm is used to perform data association in order to match the detection results with existing trackers;

[0027] When the detection result does not match the existing tracker, a new tracker is created and the motion state of the new tracker is initialized; and when any tracker is not matched within T consecutive frames, the tracker is destroyed.

[0028] Based on the detection and tracking results, the target marker and its relative position to the target vehicle are determined.

[0029] Optionally, determining the initial visual positioning information of the target vehicle based on the type of the target marker and its relative position includes:

[0030] Based on the type of the target marker and its relative position, the position information of the target marker is determined; wherein, the position information includes: marker coordinate values;

[0031] Based on the coordinate values ​​of the marker, the mapping table between coordinate values ​​and distance is consulted to obtain the distance between the target marker and the vehicle-mounted camera of the target vehicle;

[0032] The initial visual positioning information of the target vehicle is calculated based on the distance between the target marker and the vehicle-mounted camera of the target vehicle.

[0033] Optionally, the step of performing a global coordinate transformation on the initial inertial positioning information to obtain the target inertial positioning information, and performing a global coordinate transformation on the initial visual positioning information to obtain the target visual positioning information, includes:

[0034] Extract the first relative coordinates of the target vehicle from the initial inertial positioning information; based on the coordinate transformation formula, convert the first relative coordinates into the first global coordinates in the global coordinate system, which are used as the target inertial positioning information; and

[0035] The second relative coordinates of the target vehicle are extracted from the initial visual positioning information; based on the coordinate transformation formula, the second relative coordinates are converted into the second global coordinates in the global coordinate system, which are used as the target visual positioning information.

[0036] Optionally, the step of fusing the target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion information of the target vehicle includes:

[0037] An open-architecture federated Kalman filter is used to fuse the target's inertial positioning information, the target's visual positioning information, and the vehicle's ultra-wideband positioning information to obtain the vehicle's fused positioning information; wherein, the federated Kalman filter includes: multiple sub-filters and a main filter.

[0038] In the federated Kalman filter, each sub-filter includes a system state vector and a measurement vector, as well as a state transition matrix and a measurement matrix representing time k-1 to time k, for performing time updates and measurement updates to obtain state estimates and error covariance matrices;

[0039] The main filter is used to fuse the filter results of each sub-filter globally to obtain the vehicle positioning fusion location information of the target vehicle, and to predict the vehicle positioning fusion location information at future times.

[0040] This disclosure also provides an underground vehicle positioning device, including:

[0041] The acquisition module is used to acquire information about the target vehicle's electronic control unit, in-vehicle video stream, and ultra-wideband positioning information.

[0042] The first determining module is used to determine the initial inertial positioning position information of the target vehicle based on the vehicle electronic control unit information.

[0043] The second determining module is used to determine the initial visual positioning position information of the target vehicle based on the vehicle video stream information.

[0044] The conversion module is used to perform global coordinate transformation on the initial inertial positioning position information to obtain target inertial positioning position information, and to perform global coordinate transformation on the initial visual positioning position information to obtain target visual positioning position information.

[0045] The fusion module is used to fuse the target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion information of the target vehicle.

[0046] This disclosure also provides an electronic device, including:

[0047] processor;

[0048] Memory used to store the processor's executable instructions;

[0049] The processor is configured to execute the instructions to implement the steps in the above method.

[0050] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0051] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0052] In some embodiments of this disclosure, vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information of the target vehicle are acquired; based on the vehicle electronic control unit information, the initial inertial positioning position information of the target vehicle is determined; based on the in-vehicle video stream information, the initial visual positioning position information of the target vehicle is determined; the initial inertial positioning position information is subjected to global coordinate transformation to obtain the target inertial positioning position information, and the initial visual positioning position information is subjected to global coordinate transformation to obtain the target visual positioning position information; the target inertial positioning position information, the target visual positioning position information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion position information of the target vehicle; this disclosure combines vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information to achieve multi-source positioning information fusion of the target vehicle, thereby improving the positioning accuracy of the vehicle in motion.

[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0055] Figure 1 A flowchart illustrating an exemplary embodiment of this disclosure for a method of locating an underground vehicle;

[0056] Figure 2 A schematic diagram of an underground vehicle positioning system provided for an exemplary embodiment of this disclosure;

[0057] Figure 3 A flowchart illustrating a marker tracking algorithm provided for an exemplary embodiment of this disclosure;

[0058] Figure 4 A schematic diagram of a feedback-free reset federated Kalman filter provided for an exemplary embodiment of this disclosure;

[0059] Figure 5 A schematic diagram of the structure of an underground vehicle positioning device provided for an exemplary embodiment of this disclosure;

[0060] Figure 6 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0062] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0063] It should be noted that the user information involved in this disclosure includes, but is not limited to, user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0064] To address the aforementioned technical problems, in some embodiments of this disclosure, the vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information of the target vehicle are acquired; based on the vehicle electronic control unit information, the initial inertial positioning position information of the target vehicle is determined; based on the in-vehicle video stream information, the initial visual positioning position information of the target vehicle is determined; the initial inertial positioning position information is subjected to global coordinate transformation to obtain the target inertial positioning position information, and the initial visual positioning position information is subjected to global coordinate transformation to obtain the target visual positioning position information; the target inertial positioning position information, the target visual positioning position information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion position information of the target vehicle; this disclosure combines vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information to achieve multi-source positioning information fusion of the target vehicle, thereby improving the positioning accuracy of the vehicle in motion.

[0065] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0066] Figure 1This is a flowchart illustrating an exemplary embodiment of an underground vehicle positioning method provided in this disclosure. Figure 1 As shown, the method includes:

[0067] S101: Obtain vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information of the target vehicle;

[0068] S102: Determine the initial inertial positioning information of the target vehicle based on the information from the vehicle's electronic control unit;

[0069] S103: Determine the initial visual positioning information of the target vehicle based on the vehicle video stream information;

[0070] S104: Perform global coordinate transformation on the initial inertial positioning information to obtain the target inertial positioning information, and perform global coordinate transformation on the initial visual positioning information to obtain the target visual positioning information;

[0071] S105: The target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion information of the target vehicle.

[0072] In some embodiments of this disclosure, the subject executing the above method is a terminal device or a server.

[0073] The terminal device includes, but is not limited to, mobile stations (MS), mobile terminals, mobile phones, handsets, and portable equipment. This terminal device can communicate with one or more core networks via a radio access network (RAN). For example, the terminal device can be a mobile phone (or "cellular" phone), a computer with wireless communication capabilities, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an AR terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical care, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. The operating systems installed on the terminal device include, but are not limited to, iOS, Android, Windows, Linux, and Mac OS. In different networks, terminals may be called by different names, such as: user equipment, mobile station, user unit, station, cellular phone, personal digital assistant, wireless modem, wireless communication device, handheld device, laptop, cordless phone, wireless local loop station, television, etc. For ease of description, this embodiment will simply refer to it as terminal device.

[0074] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly consists of a processor, hard disk, memory, system bus, and other common computer architecture types.

[0075] In this embodiment, vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information of the target vehicle are acquired; based on the vehicle electronic control unit information, the initial inertial positioning position information of the target vehicle is determined; based on the in-vehicle video stream information, the initial visual positioning position information of the target vehicle is determined; the initial inertial positioning position information is subjected to global coordinate transformation to obtain the target inertial positioning position information, and the initial visual positioning position information is subjected to global coordinate transformation to obtain the target visual positioning position information; the target inertial positioning position information, the target visual positioning position information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion position information of the target vehicle; this disclosure combines vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information to achieve multi-source positioning information fusion of the target vehicle, thereby improving the positioning accuracy of the vehicle in motion.

[0076] Figure 2 This is a schematic diagram of an underground vehicle positioning system provided as an exemplary embodiment of this disclosure. Figure 2As shown, the vehicle positioning system includes: an inertial navigation positioning calculation module, a marker detection and tracking module, a marker recognition module, a visual positioning calculation module, a coordinate system transformation module, and a positioning fusion module. The inertial navigation positioning calculation module performs positioning calculations based on information from the vehicle's electronic control unit. It uses information such as wheel speed, acceleration, and heading angle to calculate the vehicle's inertial positioning data using the ESKF (Error State Kalman Filter) algorithm. During the positioning process, the inertial navigation module provides continuity and real-time tracking of the vehicle's position. The marker detection and tracking module detects and tracks designated markers. This module performs real-time detection and tracking of the vehicle's video stream, detecting markers appearing in the video stream and continuously tracking their relative motion. The marker recognition module identifies the markers. Based on the detection and tracking module, this module recognizes the detected marker images and provides the identified marker ID. The visual positioning calculation module calculates the vehicle's position relative to the markers based on the existing marker calibration information and visual marker recognition information, thus solving for the vehicle's visual positioning result and providing the vehicle's visual positioning position information. The coordinate system transformation module converts relative distances to absolute coordinates, typically requiring a known reference coordinate system (such as GPS or map coordinates). This module accepts distance data from the visual positioning and inertial navigation positioning modules and combines this information to convert relative positions to global coordinates. The positioning fusion and prediction module generates real-time, continuous vehicle positioning fusion information based on the vehicle's ultra-wideband positioning information, visual positioning information, and inertial positioning information, and predicts the vehicle's position after time dt. This module uses visual positioning as calibration information and inertial navigation position as continuous prediction information to achieve high-precision continuous position information output. dt represents the time delay caused by data acquisition, transmission, and processing; this module automatically compensates for positioning errors due to this delay.

[0077] This disclosure establishes a fusion-based precise positioning system based on AI visual positioning. This system eliminates the need for additional positioning base stations; instead, it integrates existing positioning system data, vehicle ECU data, etc., and achieves precise vehicle positioning through a fusion algorithm. The system includes a fusion-based precise positioning algorithm and an AI platform. The AI ​​platform is primarily used to deploy the fusion-based precise positioning algorithm. This platform provides functions such as IoT, video cloud, artificial intelligence, and device integration.

[0078] This AI-integrated precision positioning system aims to improve the real-time positioning accuracy of transport vehicles in coal mine roadways, overcoming the shortcomings of existing UWB positioning technology in dynamic environments. Based on advanced technologies such as 5G, AI, and IoT, the system comprehensively utilizes onboard video streams, vehicle ECU data, and AI visual positioning models to provide transport vehicles with dynamic and accurate real-time positioning information. By introducing inertial navigation and visual recognition algorithms, as well as a delay compensation mechanism, the system can achieve continuous vehicle positioning and prediction at a low cost under dynamic operating conditions, ensuring a positioning accuracy of no more than 7.3 meters, thereby meeting the safety and efficiency requirements of coal mine production.

[0079] The AI-based visual positioning fusion precision positioning system application receives in-vehicle video stream information and vehicle ECU information, and, based on task management of the AI ​​computing power management platform, completes accurate prediction and positioning of multiple vehicle locations. First, the vehicle's video stream information and ECU information are transmitted to the AI ​​computing power management platform via a 5G network. Then, the AI ​​fusion precision positioning application, in the form of an application mirror, is managed by the computing power scheduling management platform, and its interface must meet the requirements of the scheduling management platform. Finally, the vehicle location prediction and positioning results are sent to the intelligent human-machine interaction system and the intelligent assisted transportation management platform for displaying the vehicle location on the intelligent assisted transportation management platform.

[0080] The NTP time server provides time services to all devices within the system, synchronizing their clocks. It also corrects the system timestamps in the video stream and ECU data packets.

[0081] Leveraging the high bandwidth, low latency, high reliability, and wide connectivity of the Internet of Things (IoT) characteristics of 5G networks, this implementation scheme combines AI visual positioning with vehicle electronic control unit (ECU) information positioning for precise vehicle location assistance. By fusing these three positioning data sources, precise vehicle location is achieved both in motion and at rest. This implementation scheme has the following characteristics:

[0082] In terms of scientific rigor, this solution combines multiple positioning technologies, including UWB wireless positioning, AI visual positioning, and vehicle electronic control unit information positioning. These technologies all have their scientific basis and mature theoretical foundations. By fusing and calculating the data from these three positioning sources, the advantages of different technologies can be fully utilized to improve the accuracy and reliability of positioning, demonstrating a high level of scientific validity.

[0083] In terms of completeness, this system not only achieves accurate positioning of vehicles when stationary, but also solves the problem of excessive positioning errors during motion, meeting the comprehensive needs of coal mines for vehicle positioning. From the selection of positioning technology to data fusion algorithms, and then to process optimization in practical applications, this solution covers multiple aspects such as technology, algorithms, and applications, demonstrating high completeness.

[0084] In terms of advancement, this solution incorporates new technologies. AI visual positioning and vehicle electronic control unit (ECU) information positioning are both advanced positioning technologies. AI visual positioning utilizes computer vision technology to identify and locate vehicles, offering advantages such as high precision and real-time performance. ECU information positioning, on the other hand, obtains location information through the vehicle's own ECU, exhibiting high stability and reliability. Furthermore, this solution integrates multiple advanced technologies, achieving innovation in the positioning system and improving positioning accuracy and efficiency, thus demonstrating significant advancement.

[0085] In terms of practicality, this solution addresses the problems existing in the original UWB wireless positioning system in coal mines, proposing an effective solution that can significantly improve the accuracy and reliability of vehicle positioning, meeting the actual needs of coal mine production and management. Furthermore, the system design considers ease of operation in practical applications, using interfaces to connect with other systems, thus demonstrating high practicality.

[0086] In terms of compatibility, firstly, it is compatible with existing equipment: the system fully considers compatibility with the existing UWB wireless positioning system in the coal mine, enabling the reuse of existing equipment and reducing the cost and difficulty of system deployment. Secondly, it is compatible with other systems: the system was designed with compatibility with other management systems in the coal mine in mind, adopting standard protocols and interfaces to achieve data sharing and interaction, thereby improving the overall management level of the coal mine.

[0087] In terms of rationality, firstly, the technology selection is reasonable: based on the actual conditions and needs of the coal mine, appropriate positioning technologies were selected and rationally integrated. This ensures both positioning accuracy and reliability while considering cost and implementation difficulty. Secondly, the process design is reasonable: the system's process design is reasonable, enabling real-time collection, processing, and transmission of vehicle positioning data, thus improving positioning efficiency and accuracy.

[0088] In terms of feasibility, the system is technically feasible: the positioning technologies used are mature and highly feasible. Furthermore, the data fusion algorithm has undergone thorough verification and testing, ensuring stable system operation. Implementation is also feasible: the system implementation process takes into account the actual conditions and resource conditions of the coal mine, and has formulated a reasonable implementation plan and scheme, demonstrating high feasibility.

[0089] Regarding the effective utilization of existing resources, firstly, equipment reuse: the existing UWB wireless positioning system equipment in the coal mine was fully utilized, reducing the cost and difficulty of system deployment. Secondly, data integration and reuse: existing UWB positioning data was integrated and fused with AI visual positioning and vehicle electronic control unit information positioning data, improving the utilization value of the data.

[0090] In terms of development capabilities, the development team possesses extensive experience and technical expertise in positioning system development, enabling them to customize development based on the actual needs of coal mines and ensure system quality and performance. Furthermore, the development team has excellent after-sales service capabilities, able to promptly resolve issues that arise during system operation.

[0091] In some embodiments of this disclosure, vehicle electronic control unit (ECU) information, in-vehicle video stream information, and vehicle ultra-wideband (UWB) positioning information of the target vehicle are acquired. For example, the vehicle's speed, heading angle, acceleration, steering angle, and other motion state information are collected in real time through communication with the ECU via an on-board diagnostic (OBD) interface or a CAN bus interface; in-vehicle video stream information is collected by high-definition cameras installed at key viewing positions on the target vehicle and transmitted to the main control unit via an image transmission interface (such as LVDS or Ethernet); simultaneously, an UWB positioning module is deployed on the target vehicle, and combined with multiple UWB anchor points deployed in the environment, high-precision real-time position data of the vehicle is obtained through TOA (Time of Arrival) or TDOA (Time Difference of Arrival) positioning algorithms. Through the above methods, the system can efficiently and synchronously acquire multi-source positioning-related information of the vehicle, providing data support for subsequent positioning fusion and state estimation.

[0092] It should be noted that the vehicle electronic control unit information includes, but is not limited to: specific force, angular velocity, vehicle speed, engine status information, transmission information, braking system information, steering system information, vehicle attitude information, environmental perception information, safety system status, and on-board diagnostic information.

[0093] In some embodiments of this disclosure, the initial inertial positioning information of the target vehicle is determined based on vehicle electronic control unit (ECU) information. One possible approach is to obtain given initial conditions; input the given initial conditions, specific force, and angular velocity into a trajectory recursion algorithm to obtain the initial inertial positioning information of the target vehicle. Specifically, an autonomous positioning system utilizes vehicle ECU information (i.e., vehicle electronic control unit information) to obtain the vehicle's specific force and angular velocity information, combined with the given initial conditions, to perform real-time calculations of parameters such as speed, position, and attitude. Inertial navigation positioning calculation is a type of calculation-based navigation method, which calculates the position of the next point from the position of a known point based on continuously measured heading angles and velocities of the moving body, thus continuously measuring the current position of the moving object.

[0094] Dead Reckoning (DR) is the primary method for inertial navigation positioning calculations. The DR algorithm calculates the navigation state for the next moment based on sensor observations, given the previous navigation state (state, velocity, and position). The DR algorithm comprises attitude orchestration and position orchestration. The proposed dead reckoning algorithm uses the ESKF algorithm, short for Error State Kalman Filter. Compared to the original Kalman filter, its advantages are as follows: errors are small and second-order errors are negligible; the increment of rotation is represented by a small three-dimensional quantity, rather than a quaternion or rotation matrix; and ESKF always operates near the origin, thus avoiding singularities.

[0095] The basic flow of the trajectory recursion algorithm disclosed herein is as follows: Initialize the state variables and Kalman state variables. Here, if gravity g is included, the pose is initialized to the identity matrix I; otherwise, a reasonable initial pose R(0) needs to be determined. When ECU data arrives, inertial navigation calculations are performed on the ECU, updating the state variable x, and simultaneously updating the error state variable ax using the state transition matrix. If a return state variable is needed, x+ax can be returned, but ax=0 (it has not been updated yet, and its result will drift rapidly). When a heterogeneous observation data arrives, such as visual positioning information, the observation error state variable needs to be corrected. After obtaining the corrected ax', x+ax' can be returned.

[0096] In some embodiments of this disclosure, the initial visual positioning information of the target vehicle is determined based on the vehicle-mounted video stream information. One possible approach is to perform sign detection and sign tracking based on the vehicle-mounted video stream information to obtain the target sign and its relative position to the target vehicle; identify the target sign to obtain its type; and determine the initial visual positioning information of the target vehicle based on the type and relative position of the target sign.

[0097] In the above embodiments, sign detection and sign tracking are performed based on the vehicle video stream information to obtain the target sign and the relative position of the target sign and the target vehicle. One possible approach involves performing data augmentation on the vehicle video stream and initializing a set of anchor presets to obtain the input image for object detection; using a feature extraction network to extract features from the input image to obtain multi-scale feature maps; fusing the multi-scale feature maps using a PANet structure to obtain fused feature map information; adjusting the parameters of the object detection network based on a classification loss function and a bounding box regression loss function to predict the category and location information of the target marker; predicting the motion state of the target marker using Kalman filtering based on the detection results of the current frame and historical frames; using the Hungarian algorithm to perform data association based on the intersection-union distance between the predicted box and the detection box in the current frame to match the detection results with existing trackers; creating a new tracker and initializing the motion state of the new tracker when the detection result does not match an existing tracker; and destroying any tracker if it is not matched within T consecutive frames; and determining the target marker and its relative position to the target vehicle based on the detection and tracking results. The marker detection and tracking module is the foundation for visual positioning and consists of two main parts: detection algorithm and tracking algorithm.

[0098] For the marker detection part, to ensure the overall performance of the system, a high marker detection rate is required. Therefore, a target detection algorithm based on a deep learning network is proposed. Referring to the main idea of ​​the YOLO model, the main structure and functions of the network are described as follows: 1) Preparation (during Input): The image needs to undergo data augmentation (especially Mosaic data augmentation), and a set of anchor presets are initialized. 2) Feature extraction (during Backbone): The basic structure of Conv, C3, and SPPF is used to extract features from the input image. Conv is used to downsample the input (a total of 5 downsampling operations are performed); C3 is used to extract and fuse features from the input to enrich the semantic information of the features. In this process, Boottleneck is used to reduce the number of parameters and computation, and the idea of ​​CSPNet is borrowed to enhance the learning ability of CNN; SPPF uses pooling and feature fusion to enrich the semantic information of the features, so that the deepest feature map has extremely rich semantic information. 3) Feature processing (during Neck): Shallow features are fused between the feature maps of the three scales to be detected (shallow features are beneficial for detection). Drawing inspiration from PANet, shallow features are fused into the extracted feature maps, resulting in feature maps that possess both rich semantic information and accurate object location information. 4) Target prediction (done in the Head): Prediction is performed on the processed feature maps, and the parameter weights are optimized based on the loss function (Classification Loss and BoundingBoxRegeression Loss) and the optimizer.

[0099] For the marker tracking algorithm, a "tracking-by-detection" framework is proposed, where the tracker only needs to utilize the detection results of the current and previous frames to implement the tracking algorithm. No appearance features are used during tracking; only the position and size of the detection boxes are used for motion estimation and data association.

[0100] Figure 3 This is a flowchart illustrating a marker tracking algorithm provided for an exemplary embodiment of this disclosure. Figure 3 As shown, the algorithm mainly includes four modules: target detection module; motion prediction module; data association module; and marker establishment and destruction module.

[0101] The object detection module is the same as the aforementioned marker detection module, whose output is the position and size of the detected markers in the image. The motion prediction module, if three markers are detected in the current frame, uses methods such as Kalman filtering to predict the state of these three objects in the next frame (or several frames later). The prediction process uses a motion model; if a detection result in the next frame is associated with one of the objects, the detection result is used as the observation to update the object's state. The data association module answers which existing object the currently detected object belongs to. Assuming three objects were detected in the previous frame and four objects are detected in the current frame, a 3×4 matrix is ​​obtained. Each element in the matrix represents the IoU distance (Intersection-over-union distance) between the predicted bounding box and the currently detected bounding box. Based on this matrix, to minimize the total IoU distance, the Hungarian algorithm can be used for matching / assignment to complete the data association. The marker creation and destruction module will create a new tracker if the IoU between a detected object and all existing trackers is small. The velocity of this object will be initialized to 0, and the velocity-related covariance components will be initialized to a large value. If a tracked object is not detected and associated within T frames, the tracker will be destroyed.

[0102] In the above embodiments, target markers are identified to determine their type. The purpose of the marker identification module is to identify which marker is located within the region of interest cropped by the detection and tracking module. To improve the recognition rate, a deep learning model is proposed, referencing network structures such as FaceNet, to complete the marker identification. The training process of the identification network can be roughly divided into the following steps:

[0103] Data preprocessing: Images in the marker dataset often contain the entire marker and part of the environmental background, and often have tilting, rotation and other issues. Before inputting such images into the model, it is necessary to slice the marker part of the image, remove irrelevant background information, and align the sliced ​​marker images before they can be used for training.

[0104] Datasets are loaded according to specific rules: This system uses a new data import method: the dataset is loaded in units of two matching images (two images of the same marker) and one non-matching image (one image of another marker).

[0105] Extracting marker features using large backbone networks: By selecting a suitable deep convolutional neural network and modifying the output layer, feature extraction can be performed. Commonly used deep convolutional neural networks include the ResNet series and the Inception series.

[0106] L2 norm normalization of marker feature information: The output of the deep convolutional neural network is a feature vector in a 128-dimensional hyperspace. L2 regularization is required to normalize the points in the 128-dimensional hyperspace to a 128-dimensional hypersphere.

[0107] Loss calculation and gradient update: Using a specific "ternary loss function" can make the model quickly optimize in the direction of the target.

[0108] In the above embodiments, the initial visual positioning information of the target vehicle is determined based on the type and relative position of the target marker. One possible approach is to determine the position information of the target marker based on its type and relative position; wherein the position information includes: marker coordinate values; based on the marker coordinate values, a mapping table between coordinate values ​​and distances is consulted to obtain the distance between the target marker and the vehicle's onboard camera; based on the distance between the target marker and the vehicle's onboard camera, the initial visual positioning information of the target vehicle is calculated. The visual positioning calculation module refers to calculating the vehicle's position relative to the marker based on the existing marker-marked position information and visual marker recognition information, thereby solving for the vehicle's visual positioning result. It includes three calculation steps: querying the current marker-marked position information x; calculating the distance dx between the marker and the onboard camera based on the marker's coordinate values ​​in the image; and calculating the vehicle's position information.

[0109] In some embodiments of this disclosure, initial inertial positioning information is transformed into target inertial positioning information using global coordinate transformation, and initial visual positioning information is transformed into target visual positioning information using global coordinate transformation. One possible approach is to extract the first relative coordinates of the target vehicle from the initial inertial positioning information; based on a coordinate transformation formula, convert the first relative coordinates into first global coordinates in the global coordinate system, serving as the target inertial positioning information; and extract the second relative coordinates of the target vehicle from the initial visual positioning information; based on a coordinate transformation formula, convert the second relative coordinates into second global coordinates in the global coordinate system, serving as the target visual positioning information. The coordinate transformation module is used to convert the relative distance data of the vehicle output by the visual positioning calculation module and the inertial navigation positioning calculation module into the vehicle's position in the coordinate system. Through this module, the relative position can be combined with the known coordinates of the marker to calculate the vehicle's actual position in the coordinate system, thereby providing data support for further positioning fusion. This includes three calculation steps: obtaining relative distance information; combining the global coordinates of the marker and using an appropriate coordinate transformation formula (such as rotation matrix, translation, etc.) to convert the vehicle's relative coordinates into coordinate values ​​in the global coordinate system. This calculation takes into account the vehicle's orientation and the known positions of landmarks, ensuring that the transformation result is consistent with the global coordinate system. The calculated global vehicle coordinate information is then passed to the localization fusion prediction module, which integrates data from different sensors (such as vision, inertial navigation, etc.) to improve localization accuracy and ultimately determine the vehicle's precise position.

[0110] In some embodiments of this disclosure, the target's inertial positioning information, target's visual positioning information, and vehicle's ultra-wideband positioning information are fused to obtain the target vehicle's fused positioning information. One possible approach is to use an open-architecture federated Kalman filter to fuse the target's inertial positioning information, target's visual positioning information, and vehicle's ultra-wideband positioning information to obtain the target vehicle's fused positioning information. The federated Kalman filter includes multiple sub-filters and a main filter. Each sub-filter includes a system state vector and a measurement vector, as well as a state transition matrix and a measurement matrix representing time k-1 to time k, used to perform time updates and measurement updates to obtain state estimates and error covariance matrices. The main filter is used to fuse the filter results of each sub-filter globally to obtain the target vehicle's fused positioning information and predict the vehicle's fused positioning information at future times. The positioning fusion prediction module generates real-time, continuous vehicle positioning fusion location information based on ultra-wideband vehicle positioning information, visual positioning location information, and inertial positioning location information. It can also predict positioning fusion errors caused by information delays and automatically compensate for and predict the vehicle's current position. To improve the fault tolerance and information source expansion capabilities of the precise positioning system, a federated Kalman filter is designed, employing an open algorithm architecture and selecting typical vehicle positioning information sources for fusion. The algorithm framework uses a common reference system as a benchmark, forming sub-filters with other information sources. The outputs of the sub-filters, after fault diagnosis and system reconstruction, enter the main filter for information fusion.

[0111] Figure 4 This is a schematic diagram of a feedback-free reset federated Kalman filter provided as an exemplary embodiment of this disclosure. Figure 4 As shown, each local filter is independent of the others, and there is no mutual influence caused by feedback reset, which provides the highest fault tolerance. The corresponding discretized system error model is as follows:

[0112]

[0113] In the formula, Let i be the system state vector of the i-th sub-filter; Let be the measurement vector of the i-th sub-filter; Φk|k-1 represents the one-step state transition matrix from time k-1 to time k; The measurement matrix is ​​defined. Each of the i sub-filters independently performs time and measurement updates to obtain the state estimate. And error covariance matrix

[0114] When performing time recursion in the global filter, it is necessary to predict the fused position information of vehicles at future times. When the system completes the time update, it needs to calculate the difference between the current algorithm platform clock and the timestamp of each data packet, dt = Tt, where T represents the current algorithm platform time and t represents the timestamp of the current data packet. Based on the fused data at time t, it needs to complete the vehicle position prediction for the current time T = t + dt, and complete the time delay position compensation.

[0115] Through the processes of time update, measurement update, information fusion, and information allocation, the sub-filters lose some information during noise allocation, resulting in suboptimal filtering results. However, during the fusion of the sub-filter results composed of various information sources, the non-optimal aspects are resynthesized to obtain the globally optimal estimate.

[0116] The main functions disclosed herein include: high-precision positioning, safety warning and alarm, trajectory recording and backtracking, data storage and querying, and vehicle map generation. Specifically: High-precision positioning: The system provides high-precision vehicle positioning services through the existing UWB positioning system and newly added AI multi-element fusion and latency compensation functions. In a static environment, the positioning accuracy can reach within 0.3 meters; during dynamic driving, the positioning accuracy does not exceed 7.3 meters, meeting the positioning needs of coal mine transport vehicles in complex tunnel environments. Safety warning and alarm: The system monitors the relative distances between vehicles and between vehicles and personnel in real time. When the distance between vehicles is less than 150 meters, or the distance between a vehicle and personnel is less than 100 meters, the system will automatically trigger a safety distance warning to remind drivers and relevant personnel, ensuring the safety of coal mine operations. Trajectory recording and backtracking: The system provides an interface for recording real-time vehicle trajectory and location information, allowing the system to record vehicle operating trajectory data and supporting subsequent trajectory backtracking, driving analysis, and accident tracking. Simultaneously, this data can be used to generate vehicle driving maps, providing management with comprehensive analysis support for vehicle operation within tunnels. The system's data storage and retrieval functions store vehicle location and trajectory data in real time, facilitating the retrieval of historical vehicle location records at any time. This provides support for historical data analysis of vehicle operations and assists in the development of optimization plans. These functions together constitute the core module of the system, ensuring the safe and efficient operation of coal mine transport vehicles and providing reliable data support for management decisions.

[0117] This disclosure fully utilizes existing UWB positioning systems and camera equipment, eliminating the need for additional positioning base stations. This not only saves on equipment procurement and installation costs but also shortens system deployment time. Simultaneously, leveraging existing infrastructure ensures system compatibility and stability, reducing the technical risks and maintenance complexities that might arise from introducing new equipment. By optimizing the use of existing resources, the project achieves efficient resource integration, improving the overall system's economic benefits.

[0118] High-precision positioning in both static and dynamic environments. In a static vehicle state, based on UWB positioning technology, the system ensures a positioning accuracy within 0.3 meters, meeting the requirements for high-precision static positioning in complex environments such as coal mines. In a dynamic vehicle state, the system employs AI multi-source fusion technology, including target detection and recognition positioning, camera vision assistance, and inertial navigation calculations using ECU data, to achieve precise vehicle positioning. Through multi-source information fusion, a positioning accuracy of 7.3 meters is achieved in dynamic states, ensuring the continuity and accuracy of positioning and providing reliable support for complex dynamic application scenarios.

[0119] Multi-source fusion of AI vision and inertial navigation data. AI vision technology is used to perform target detection on data collected by cameras, combined with inertial navigation positioning based on vehicle acceleration and speed information collected by the ECU, constructing a multi-source information fusion positioning scheme. This technology effectively improves the accuracy and stability of positioning in dynamic environments, solving the problem of accuracy degradation of single positioning technologies in dynamic scenarios, thus meeting the requirements for dynamic and precise positioning while ensuring system robustness.

[0120] A latency compensation mechanism ensures positioning accuracy. The system design incorporates a latency compensation mechanism to guarantee the real-time performance of data transmission and processing. Due to potential delays during network transmission and data processing, the system uses a latency compensation algorithm to correct the positioning data, ensuring the accuracy and synchronization of vehicle location information. Combined with AI multi-element fusion technology, the latency compensation mechanism effectively reduces the impact of transmission delays on positioning results, providing more accurate vehicle location data for real-time monitoring and improving positioning reliability and response speed.

[0121] Flexible system scalability. The system boasts excellent scalability by independently processing vehicle data using application mirroring. As the coal mine expands and the number of managed vehicles increases, the number of vehicles can be expanded simply by increasing server computing power and adding application mirrors. It also offers scalability for other data sources. The system design provides multiple data interfaces for easy integration with other data sources, such as sensor data, surveillance video streams, and GPS data, supporting seamless integration with existing systems. Through modular and standardized interface design, new data source integration can be easily achieved via configuration files or plug-ins. This allows the system to not only adapt to the growth of vehicle data but also further enhance its functionality and intelligence by integrating more data sources, meeting ever-changing business needs.

[0122] In the entire vehicle positioning system, the data flow begins from multiple input modules and achieves precise vehicle positioning through multiple algorithm processing modules. First, it receives dynamic data (such as wheel speed, acceleration, and steering angle) from the vehicle's ECU and video stream data from the onboard camera. ECU data is used for inertial navigation positioning, while the video stream is used to detect surrounding landmarks. In the algorithm processing module, the inertial navigation positioning calculation module receives the ECU data and uses the Error State Kalman Filter (ESKF) algorithm to calculate the vehicle's inertial navigation positioning data, ensuring the continuity and real-time nature of the vehicle's position even when landmarks are obscured or not present in the video image. Next, the landmark detection and tracking module detects and tracks preset landmarks in the video stream data, identifies the relative positions of these landmarks to the vehicle, and transmits the detection results to the next module. The landmark recognition module assigns a unique ID to each detected landmark, ensuring the system can distinguish between different landmarks, and transmits the recognition results to the visual positioning module. In the visual positioning calculation module, combining the landmark's identification ID and known position, the precise position of the vehicle relative to the landmark is calculated, generating visual positioning data. Both inertial navigation (INS) and visual positioning data are fed into a coordinate transformation module to convert the relative positions into accurate coordinate system positions. Finally, in the positioning fusion and prediction module, the transformed INS and visual positioning data are fused to generate high-precision real-time vehicle positioning information. This module also performs latency compensation to eliminate delays in data processing and transmission, and uses INS information to predict the vehicle's future position in the short term, ensuring continuous data output. Through multi-source data fusion, positioning algorithms, and latency compensation mechanisms, the entire system can provide high-precision vehicle positioning services in complex environments.

[0123] Figure 5 This is a schematic diagram of the structure of an underground vehicle positioning device 50 provided for an exemplary embodiment of this disclosure. Figure 5 As shown, the underground vehicle positioning device 50 includes: an acquisition module 51, a first determination module 52, a second determination module 53, a conversion module 54, and a fusion module 55.

[0124] Among them, the acquisition module 51 is used to acquire the vehicle electronic control unit information, vehicle video stream information and vehicle ultra-wideband positioning information of the target vehicle.

[0125] The first determining module 52 is used to determine the initial inertial positioning position information of the target vehicle based on the vehicle electronic control unit information.

[0126] The second determining module 53 is used to determine the initial visual positioning position information of the target vehicle based on the vehicle video stream information.

[0127] The conversion module 54 is used to perform global coordinate transformation on the initial inertial positioning position information to obtain the target inertial positioning position information, and to perform global coordinate transformation on the initial visual positioning position information to obtain the target visual positioning position information.

[0128] The fusion module 55 is used to fuse the target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion information of the target vehicle.

[0129] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0130] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. For example... Figure 6 As shown, the electronic device includes a memory 61 and a processor 62. Additionally, the electronic device also includes a power supply component 63 and a communication component 64.

[0131] Memory 61 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0132] The memory 61 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0133] Communication component 64 is used for data transmission with other devices.

[0134] The processor 62 is capable of executing computer instructions stored in the memory 61 to: acquire vehicle electronic control unit information, in-vehicle video stream information, and vehicle ultra-wideband positioning information of the target vehicle; determine the initial inertial positioning position information of the target vehicle based on the vehicle electronic control unit information; determine the initial visual positioning position information of the target vehicle based on the in-vehicle video stream information; perform global coordinate transformation on the initial inertial positioning position information to obtain the target inertial positioning position information, and perform global coordinate transformation on the initial visual positioning position information to obtain the target visual positioning position information; and fuse the target inertial positioning position information, the target visual positioning position information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion position information of the target vehicle.

[0135] Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores a computer program, and the computer program is executed by one or more processors, it causes one or more processors to perform... Figure 1 Each step in the method embodiment.

[0136] Accordingly, embodiments of this disclosure also provide a computer program product, which includes a computer program / instructions that are executed by a processor. Figure 1 Each step in the method embodiment.

[0137] The above Figure 6 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0138] The above Figure 6 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0139] The aforementioned electronic devices also include a display screen and audio components.

[0140] The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0141] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0142] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0150] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating underground vehicles, characterized in that, include: Acquire information about the target vehicle's electronic control unit, in-vehicle video stream, and ultra-wideband positioning. Based on the vehicle electronic control unit information, the initial inertial positioning position information of the target vehicle is determined; Based on the in-vehicle video stream information, the initial visual positioning location information of the target vehicle is determined; The initial inertial positioning information is transformed into global coordinates to obtain the target inertial positioning information, and the initial visual positioning information is transformed into global coordinates to obtain the target visual positioning information. The target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information are fused to obtain the vehicle positioning fusion location information of the target vehicle; wherein... Determining the initial visual positioning information of the target vehicle based on the in-vehicle video stream information includes: The in-vehicle video stream information is subjected to data augmentation processing, and a set of anchor presets is initialized to obtain an input image for target detection; The input image is used to extract features using a feature extraction network to obtain a multi-scale feature map. The multi-scale feature maps are fused using the PANet structure to obtain fused feature map information; Based on the classification loss function and the bounding box regression loss function, the parameters of the target detection network are adjusted by an optimizer to predict the category and location information of the target marker; Based on the detection results of the current frame and historical frames, the motion state of the target marker is predicted using Kalman filtering; Based on the intersection-union distance between the predicted bounding box and the detection bounding box in the current frame, the Hungarian algorithm is used to perform data association in order to match the detection results with existing trackers; When the detection result does not match the existing tracker, a new tracker is created and the motion state of the new tracker is initialized; and when any tracker is not matched within T consecutive frames, the tracker is destroyed. Based on the detection and tracking results, the target marker and its relative position to the target vehicle are determined. The target marker is identified to determine its type; The initial visual positioning information of the target vehicle is determined based on the type of the target marker and its relative position.

2. The method according to claim 1, characterized in that, The vehicle electronic control unit information includes: specific force and angular velocity; determining the initial inertial positioning information of the target vehicle based on the vehicle electronic control unit information includes: Obtain the given initial conditions; The given initial conditions, the specific force, and the angular velocity are input into the trajectory recursion algorithm to obtain the initial inertial positioning information of the target vehicle.

3. The method according to claim 1, characterized in that, Determining the initial visual positioning information of the target vehicle based on the type of the target marker and its relative position includes: Based on the type of the target marker and its relative position, the position information of the target marker is determined; wherein, the position information includes: marker coordinate values; Based on the coordinate values ​​of the marker, the mapping table between coordinate values ​​and distance is consulted to obtain the distance between the target marker and the vehicle-mounted camera of the target vehicle; The initial visual positioning information of the target vehicle is calculated based on the distance between the target marker and the vehicle-mounted camera of the target vehicle.

4. The method according to claim 1, characterized in that, The step of performing global coordinate transformation on the initial inertial positioning information to obtain target inertial positioning information, and performing global coordinate transformation on the initial visual positioning information to obtain target visual positioning information, includes: Extract the first relative coordinates of the target vehicle from the initial inertial positioning information; based on the coordinate transformation formula, convert the first relative coordinates into the first global coordinates in the global coordinate system, which are used as the target inertial positioning information; and The second relative coordinates of the target vehicle are extracted from the initial visual positioning information; based on the coordinate transformation formula, the second relative coordinates are converted into the second global coordinates in the global coordinate system, which are used as the target visual positioning information.

5. The method according to claim 1, characterized in that, The step of fusing the target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion information of the target vehicle includes: An open-architecture federated Kalman filter is used to fuse the target's inertial positioning information, the target's visual positioning information, and the vehicle's ultra-wideband positioning information to obtain the vehicle's fused positioning information; wherein, the federated Kalman filter includes: multiple sub-filters and a main filter. In the federated Kalman filter, each sub-filter includes a system state vector and a measurement vector, as well as a state transition matrix and a measurement matrix representing time k-1 to time k, for performing time updates and measurement updates to obtain state estimates and error covariance matrices; The main filter is used to fuse the filter results of each of the sub-filters into global information to obtain the vehicle positioning fusion location information of the target vehicle, and to predict the vehicle positioning fusion location information at future times.

6. A positioning device for underground vehicles, characterized in that, include: The acquisition module is used to acquire information about the target vehicle's electronic control unit, in-vehicle video stream, and ultra-wideband positioning information. The first determining module is used to determine the initial inertial positioning position information of the target vehicle based on the vehicle electronic control unit information. The second determining module is used to determine the initial visual positioning information of the target vehicle based on the vehicle video stream information. Specifically, the vehicle video stream information undergoes data augmentation processing, and a set of anchor presets is initialized to obtain an input image for target detection. A feature extraction network is used to extract features from the input image to obtain multi-scale feature maps. A PANet structure is used to fuse the multi-scale feature maps to obtain fused feature map information. Based on a classification loss function and a bounding box regression loss function, the parameters of the target detection network are adjusted by an optimizer to predict the category and location information of the target marker. Based on the detection results of the current frame and historical frames, Kalman filtering is used to determine the motion of the target marker. The system predicts the state; based on the intersection-over-union (IoU) distance between the predicted bounding box and the detection box in the current frame, it uses the Hungarian algorithm to perform data association to match the detection results with existing trackers; when the detection results do not match with existing trackers, a new tracker is created and the motion state of the new tracker is initialized; and when any tracker is not matched within T consecutive frames, the tracker is destroyed; based on the detection results and tracking results, the target marker and its relative position to the target vehicle are determined; the target marker is identified to obtain its type; and based on the type of the target marker and its relative position, the initial visual positioning information of the target vehicle is determined. The conversion module is used to perform global coordinate transformation on the initial inertial positioning position information to obtain target inertial positioning position information, and to perform global coordinate transformation on the initial visual positioning position information to obtain target visual positioning position information. The fusion module is used to fuse the target inertial positioning information, the target visual positioning information, and the vehicle ultra-wideband positioning information to obtain the vehicle positioning fusion information of the target vehicle.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the steps of the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

Images