Real-time Navigation Method and System for Indoor Logistics Robots with Enhanced Passive Positioning of QR Codes

Abstract

The present invention discloses a real-time navigation method and system for indoor logistics robots with enhanced passive positioning of QR codes. The method includes: collecting a small-scale QR code image data set, performing data augmentation processing to obtain an expanded data set, and dividing the expanded data set into a training set and a test set; training the weights of the TinyQR_YOLOV5 model based on the training set and the test set to obtain a QR code classifier, and determining the pixel trajectory of the robot corresponding to the video data collected in real time based on the QR code classifier; determining the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and outputting the motion trajectory of the robot. The present invention can decode QR code images at different angles more quickly, accurately and orderly, obtain the pixel trajectory of the robot at the same time, and is conducive to converting the pixel coordinates detected in real time into real coordinates, realizing passive trajectory tracking based on the camera and geographical unification.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning technology, and in particular to a real-time navigation method and system for an indoor logistics robot enhanced by two-dimensional code passive positioning. Background Art

[0002] At present, unmanned logistics robots have been used more and more widely. Unmanned logistics robots in outdoor environments mainly use the Global Navigation Satellite System (GNSS) to achieve robot positioning, but due to its generally low accuracy (5-30m) and the influence of signal attenuation in the non-line-of-sight (NLOS) transmission channel between the satellite and the receiver, the GNSS method cannot meet the positioning needs of logistics robots in indoor and underground environments. The last mile problem of unmanned logistics distribution still needs to be solved. In recent years, with the large-scale development of indoor and underground spaces in cities, the demand for unmanned logistics distribution in these environments has grown rapidly, and its social and commercial potential has attracted widespread attention.

[0003] Positioning technology for indoor environments has become a hot research topic in recent years. However, autonomous indoor positioning and navigation of robots still face problems such as low stability, high cost, and weak environmental adaptability. At present, positioning methods based on wireless signals such as Wi-Fi, Bluetooth, RFID, and UWB mainly measure spatial distances through predetermined fingerprint maps or signal transmission models to achieve indoor positioning. However, these methods have a large workload for data collection and updating, and the signal strength is easily affected by multipath transmission effects. In complex indoor environments, the positioning accuracy is easily affected by the environment, the stability is weak, and the cost is relatively high.

[0004] As a highly autonomous positioning technology, inertial positioning has been widely used in the field of robot or unmanned system positioning. Inertial positioning technology continuously calculates the robot's position information through dead reckoning, and can obtain high positioning accuracy and estimate the robot's posture in a short time. However, the positioning error of inertial positioning tends to accumulate quickly, and the effective working time is short. It is necessary to use external information to continuously calibrate the inertial sensor state parameters and positioning errors. Therefore, inertial positioning usually needs to be used together with other positioning technologies to get rid of inertial positioning errors.

[0005] As a common coding technology, QR code has been widely used in many fields, with advantages such as low cost and high versatility. At present, there are studies that use the camera on the robot to identify the QR code in the environment, so as to correct the position of the robot. For large indoor and underground scenes, these technologies need to lay out a large number of QR codes in the environment, which has high requirements for the environment layout. At the same time, the frequency of robot position correction is greatly affected by the density of QR code layout, and the stability of positioning performance is weak.

[0006] Visual positioning technology is a technology that uses cameras to obtain environmental visual information for positioning. It has the advantages of low cost, light weight, and relatively high accuracy. According to the deployment method of the camera, visual positioning technology has two modes: active and passive. Among them, active visual positioning is to mount the camera on the robot and use the camera to collect data and match it with the environmental data to calculate the position of the robot (such as visual SLAM technology). However, the positioning performance of visual SLAM positioning technology is easily affected by conditions such as environmental texture, light, and shadow, and a large amount of prior image data is required to improve the positioning accuracy, resulting in a large amount of data collection and model training work. Passive visual positioning is to use cameras installed in the environment to observe and identify the robot, and estimate the movement position of the robot based on the known camera positions. However, for large indoor or underground spaces, there are usually blind spots in the field of view of visual cameras (such as surveillance cameras). If it is necessary to generate a complete trajectory of the target in the entire surveillance camera network, it is necessary to associate the trajectory segments of the same target under different surveillance cameras and compensate for the missing trajectories. However, due to the similarity of the appearance of robots and environmental interference, there are usually errors in the appearance recognition and matching of robots based on computer vision, which has a great impact on the continuous positioning performance of robots. At present, there is still a lack of accurate and stable passive positioning solutions for logistics robots.

[0007] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a real-time navigation method and system for indoor logistics robots with enhanced passive positioning of two-dimensional codes in view of the above-mentioned defects of the existing technology, aiming to solve the problem that there is still a lack of accurate and stable passive positioning solutions for logistics robots at present.

[0009] In the first aspect, the present invention provides a real-time navigation method for indoor logistics robots with enhanced passive positioning of two-dimensional codes. Among them, the method includes:

[0010] Collect a small-scale two-dimensional code image data set, perform data enhancement processing based on the generative adversarial network to obtain an augmented data set, and divide the augmented data set into a training set and a test set;

[0011] Based on the training set and the test set, train the weights of the TinyQR_YOLOV5 model to obtain the two-dimensional code classifier of the TinyQR_YOLOV5 model, and determine the robot pixel trajectory corresponding to the video data collected in real time based on the two-dimensional code classifier;

[0012] Determine the trajectory conversion relationship between the two-dimensional code and the robot in the multi-camera network through the multi-camera perspective coordinate transformation model, and output the movement trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship.

[0013] In one implementation, the small-scale QR code image dataset is collected, enhanced based on a generative adversarial network to obtain an augmented dataset, and the augmented dataset is divided into a training set and a test set, including:

[0014] Collect the small-scale QR code image dataset by simulating the angles and distances of actual indoor application scenarios. Among them, the small-scale QR code image dataset contains QR code images with different indoor backgrounds, different scales, different angles, different visibilities, and different types;

[0015] Process the small-scale QR code image dataset through manual labeling to obtain the processed QR code image data and the label file corresponding to each image;

[0016] Enhance the processed QR code image data based on the generative adversarial network to obtain an augmented dataset.

[0017] In one implementation, the data enhancement processing includes performing affine transformation on the QR code while keeping the background unchanged, changing the type of the QR code, generating a partially occluded QR code, and replacing the background while keeping the QR code unchanged.

[0018] In one implementation, the TinyQR_YOLOV5 model includes a small object detection layer. The small object detection layer uses small-sized anchors to constrain the predicted object range, incorporates size prior experience, deepens the upsampling process of the feature map, and performs detection after splicing the shallower feature map and the deeper feature map;

[0019] Moreover, a self-attention mechanism is added to each convolutional layer in the TinyQR_YOLOV5 model.

[0020] In one implementation, determining the robot pixel trajectory corresponding to the video data collected in real time based on the QR code classifier includes:

[0021] Collect the video data in real time through an inherent camera in the scene, and input the video data into the QR code classifier in the TinyQR_YOLOV5 model to obtain the QR code border length and the central pixel coordinates respectively;

[0022] Based on the QR code border length and the central pixel coordinates, decode the image in the QR code area to obtain the information carried by the QR code and locate the robot pixel trajectory.

[0023] In one implementation, determining the trajectory conversion relationship between the QR code and the robot in the multi-camera network through the multi-camera perspective coordinate transformation model, and based on the robot pixel trajectory and the trajectory conversion relationship, outputting the motion trajectory of the robot, includes:

[0024] Based on the multi-camera perspective coordinate transformation model, calculating the covariance matrix of the centralized homologous point data set between multiple cameras, and performing singular value decomposition on the covariance matrix to determine the optimal solution;

[0025] Through iterative calculations between each camera coordinate system, obtaining the trajectory conversion relationship of the robot in the multi-camera;

[0026] Based on the trajectory conversion relationship, projecting the robot inertial navigation data into the same coordinate system to obtain the geographical coordinate conversion of different sensors;

[0027] Based on the geographical coordinate conversion and the robot pixel trajectory, obtaining the motion trajectory of the robot.

[0028] In one implementation, the method further includes:

[0029] Based on the heading of the motion trajectory of the robot, calibrating the horizontal drift of the inertia-based positioning, and constructing a visual inertial collaborative positioning scheme.

[0030] In a second aspect, an embodiment of the present invention further provides a real-time navigation system for an indoor logistics robot with enhanced passive positioning of QR codes. Among them, the system includes:

[0031] A data processing module, configured to collect a small-scale QR code image data set, perform data enhancement processing based on a generative adversarial network to obtain an augmented data set, and divide the augmented data set into a training set and a test set;

[0032] A data analysis module, configured to perform weight training on the TinyQR_YOLOV5 model based on the training set and the test set to obtain a QR code classifier of the TinyQR_YOLOV5 model, and determine the robot pixel trajectory corresponding to the video data collected in real time based on the QR code classifier;

[0033] A trajectory determination module, configured to determine the trajectory conversion relationship between the QR code and the robot in the multi-camera network through the multi-camera perspective coordinate transformation model, and based on the robot pixel trajectory and the trajectory conversion relationship, output the motion trajectory of the robot.

[0034] In a third aspect, an embodiment of the present invention further provides a terminal device, where the terminal device includes a memory, a processor, and a real-time navigation program for indoor logistics robots with enhanced passive QR code positioning stored in the memory and executable on the processor. When the processor executes the real-time navigation program for indoor logistics robots with enhanced passive QR code positioning, the steps of the real-time navigation method for indoor logistics robots with enhanced passive QR code positioning described in any one of the above solutions are implemented.

[0035] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a real-time navigation program for indoor logistics robots with enhanced passive QR code positioning is stored on the computer-readable storage medium. When the real-time navigation program for indoor logistics robots with enhanced passive QR code positioning is executed by a processor, the steps of the real-time navigation method for indoor logistics robots with enhanced passive QR code positioning described in any one of the above solutions are implemented.

[0036] Beneficial effects: Compared with the prior art, the present invention provides a real-time navigation method for indoor logistics robots with enhanced passive QR code positioning. First, the present invention collects a small-scale QR code image data set, performs data enhancement processing based on a generative adversarial network to obtain an augmented data set, and divides the augmented data set into a training set and a test set. Then, based on the training set and the test set, weight training is performed on the TinyQR_YOLOV5 model to obtain a QR code classifier of the TinyQR_YOLOV5 model, and based on the QR code classifier, the robot pixel trajectory corresponding to the video data collected in real time is determined. Finally, through a multi-camera perspective coordinate transformation model, the trajectory conversion relationship between the QR code and the robot in the multi-camera network is determined, and based on the robot pixel trajectory and the trajectory conversion relationship, the motion trajectory of the robot is output. The present invention can decode QR code images at different angles more quickly, accurately, and orderly, obtain the robot pixel trajectory at the same time, and is conducive to converting the pixel coordinates detected in real time into real coordinates, realizing passive trajectory tracking based on the camera and geographical unity. Description of the Drawings

[0037] Figure 1 It is a flowchart of the specific implementation manner of the real-time navigation method for indoor logistics robots with enhanced passive QR code positioning provided by the embodiment of the present invention.

[0038] Figure 2 It is a specific application flowchart of the real-time navigation method for indoor logistics robots with enhanced passive QR code positioning provided by the embodiment of the present invention.

[0039] Figure 3 It is a flowchart of the active and passive matching based on feature sequences in the real-time navigation method for indoor logistics robots with enhanced passive QR code positioning provided by the embodiment of the present invention.

[0040] Figure 4 This is the functional schematic diagram of the real-time navigation system for an indoor logistics robot with enhanced passive QR code positioning according to an embodiment of the present invention.

[0041] Figure 5 This is the principle block diagram of the terminal device provided by the embodiment of the present invention. Detailed implementation manners

[0042] To make the objectives, technical solutions and effects of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] This embodiment discloses a real-time navigation method for an indoor logistics robot with enhanced passive QR code positioning. The method of this embodiment uses a TinyQR_YOLOV5 locator that pays more attention to small-scale features to identify QR codes and extract the pixel trajectory of the robot, and then realizes the three-dimensional positioning of the QR code and the robot in a multi-camera network through a multi-camera perspective coordinate transformation model. The content of the scanned QR code and the feature sequence of the visual-inertial trajectory will be used to realize the association and pairing of multi-source sensors, ensuring the detection and matching effect of the active and passive methods in a multi-camera and multi-target environment. Specifically in application, the method of this embodiment can be applied to a terminal device, and the terminal device can be an intelligent product terminal such as a computer, a mobile phone, a smart TV, etc. As Figure 1 shown in, the real-time navigation method for an indoor logistics robot with enhanced passive QR code positioning includes the following steps:

[0044] Step S100: First, collect a small-scale QR code image dataset, perform data augmentation processing based on a generative adversarial network to obtain an augmented dataset, and divide the augmented dataset into a training set and a test set.

[0045] Step S200: Perform weight training on the TinyQR_YOLOV5 model based on the training set and the test set to obtain a QR code classifier of the TinyQR_YOLOV5 model, and determine the pixel trajectory of the robot corresponding to the video data collected in real time based on the QR code classifier.

[0046] Step S300: Determine the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and output the motion trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship.

[0047] Specifically, the prerequisite for the mutual calibration of passive vision-inertial sensors is to associate the same identity positioning object and establish a correct communication network. However, when the camera establishes a link through the network, it will traverse all the inertial identification marks around, and in this process, situations such as incorrect linking and interruption are likely to occur. Due to its advantages such as high information capacity, convenient and flexible use, and low deployment cost (no additional dedicated software and hardware), the two-dimensional code is a widely used information distribution method in various industries, and only a camera is required to read the content of the two-dimensional code. This embodiment proposes to achieve association and pairing through the two-dimensional code. In the scenario, the monitoring camera obtains its basic information by identifying the two-dimensional code on the robot display (or printed mark) and establishes a communication association through the wireless network to complete the matching of data sources between multiple cameras and between the camera and the inertial navigation. In the data field of the two-dimensional code in this patent, the identification information of the affiliated robot is also associated, including: the unique identification ID of the robot, the size of the robot, the type of the robot, and other parameter configurations. It should be emphasized that since the current vision algorithm cannot detect small-scale two-dimensional codes and relatively large-scale robots with high precision at the same time. Therefore, this embodiment does not obtain the detection frame of the robot. This patent obtains the size information of the robot by detecting the two-dimensional code through vision, and constructs a proportional relationship with the size of the two-dimensional code to inversely calculate the pixel coordinates of the robot. The two-dimensional code is placed in the middle of the robot, and its center coincides with the center of the robot. The pixel coordinates are assumed to be (u q , v q ). The center coordinate (u r , v r ) at the bottom of the robot can be expressed as:

[0048]

[0049] Among them, h q represents the pixel length of the side of the two-dimensional code (the two-dimensional code is square), ω is the ratio of the actual size h R of the robot to the actual size h Q of the two-dimensional code. When the camera recognizes and locates the two-dimensional code of the robot, it can obtain parameters such as the actual scales of the robot and the two-dimensional code and obtain the pixel trajectory of the robot. The encoding and decoding of the two-dimensional code are implemented through the built-in module of Open-cv to ensure the uniqueness of the two-dimensional code for matching. The devices establishing the association will achieve real-time transmission and intercommunication between the camera and the robot through the wireless communication module, correct the inertial navigation offset according to the relatively stable visual positioning result, and enable the inertial navigation autonomous positioning module to obtain a more accurate trajectory. The detailed detection and positioning method will be elaborated one by one later.

[0050] Different from the ordinary QR code scanning and positioning, the QR code images detected by the monitoring camera in complex indoor scenarios are often small in scale. At the same time, the black-and-white QR codes lack the appearance information required to distinguish them from the background or similar categories. The original detectors perform well on medium and large objects, but they perform poorly in the task of detecting QR code objects. Therefore, based on the YOLO backbone network, this embodiment proposes an identification method specifically for small-scale QR codes: the TinyQR_YOLOV5 model and the indoor scene small-scale QR code dataset. Aiming at the problem that the detection network performs poorly on small-scale targets, this embodiment improves the TinyQR_YOLOV5 model to make it suitable for extracting QR codes on moving targets from the perspective of the monitoring camera. First, this embodiment adds small target detection to the TinyQR_YOLOV5 model, that is, the predicted object range is constrained by smaller anchors, and size prior experience is added, and operations such as upsampling of the feature map are deepened, and detection is performed after splicing the shallower feature map and the deeper feature map, so as to achieve the purpose of multi-scale learning. Secondly, this embodiment adds a self-attention mechanism to each convolutional layer in the TinyQR_YOLOV5 model. After local attention operations, the input image features are converted into new features. Each such feature layer will capture useful information, such as simple QR code edges, shapes, etc., to obtain a more complex input semantic representation. The improved TinyQR_YOLOV5 model can pay more attention to the important parts of these feature maps. At the same time, the overhead is not large, and it can be trained end-to-end with the basic CNN network. The target carrying the QR code is photographed by the inherent camera in the scene, and the target and QR code detection frames are obtained through the trained weights.

[0051] Specifically, as Figure 2As shown, in this embodiment, small-scale QR code image datasets are collected by simulating the angles and distances of actual indoor application scenarios for shooting. Among them, the small-scale QR code image datasets include QR code images of different indoor backgrounds, different scales, different angles, different visibility, and different types. Then, the small-scale QR code image datasets are processed by manual marking to obtain the processed QR code image data and the label files corresponding to each image. Next, based on the generative adversarial network (GAN), data augmentation processing is performed on the processed QR code image data to obtain an augmented dataset. The data augmentation processing includes performing affine transformation on the QR code while keeping the background unchanged, changing the type of QR code, generating partially occluded QR codes, and replacing the background fusion, color transformation, contrast transformation, adding noise, etc. while keeping the QR code unchanged. The original captured pictures are recombined in the above way to increase the dimension and complexity of the dataset. Next, in this embodiment, the augmented dataset is divided into a training set and a test set, and the training set is input into the TinyQR_YOLOV5 model for weight training to obtain a trained QR code classifier. Then, the weights that perform well in multiple scenarios are selected through the test set as the QR code classifier of the TinyQR_YOLOV5 model. Next, in this embodiment, the video data is collected in real time by the camera inherent in the scenario, and the video data is input into the QR code classifier in the TinyQR_YOLOV5 model to obtain the QR code border length and the central pixel coordinates respectively. Then, based on the QR code border length and the central pixel coordinates, the image in the QR code area is decoded to obtain the information carried by the QR code and locate the pixel trajectory of the robot.

[0052] Further, the multi-camera perspective coordinate transformation model of this embodiment is:

[0053]

[0054] where (u, v, 0) are the homogeneous coordinates of the trajectory pixels in each original frame; (c x , c y ) represents the main point offset of the calibration image plane optical axis in the two-dimensional plane coordinate system; (dx, dy) respectively represent the calibration coefficient values measured in horizontal and vertical pixel units. M ∈ R 3×4 is a transformation matrix with m ij elements, which establishes the mapping relationship between the world coordinates (X i , Y i , Z i ) and the image points (u, v) based on the camera position, and is composed of the single-camera essential matrix, rotation matrix, and translation matrix. Ignoring the terrain undulation, the camera image plane is perpendicular to the plane in the world coordinate system. The world coordinates of the single camera can be expressed as:

[0055]

[0056] To solve the three-dimensional coordinate transformation relationship between multiple single cameras, this embodiment describes the problem as finding the optimal transformation matrix (R o , t o ) of the corresponding point sets (Q, P) between the cameras. Since the origin coordinates of the two world coordinate systems belong to translation, t can be temporarily eliminated by unifying the centroids of the two coordinate systems o . Therefore, the transformation task can be expressed as the minimum value of the distance of the corresponding point sets (Q′, P′) after decentalization:

[0057] (R o , t o ) = argmin(||(R o (Q′, P′))|| 2 ) (4)

[0058] t o = Q′ - R o ×P′ (5)

[0059] At this time, the optimal transformation matrix (R o , t o ) is equivalent to solving the extreme value (R o , t o ) = Tr(R o ((Q′, P′))′), calculating the covariance matrix of the centered corresponding point data set Then, performing singular value decomposition (SVD) on H o can be expressed as:

[0060] [U o , S o , V o = SVD(H o ) (6)

[0061] When the rotation matrix is , the optimal solution can be obtained. Through iterative calculations between the camera coordinate systems, the trajectory transformation relationship of the robot in multiple cameras can be obtained. Finally, the trajectory obtained through passive vision is recorded as:

[0062]

[0063] Traj pa represents the global trajectory of object passive positioning, and traj i is the position data of the i-th object, where t is the timestamp, I is the data information encoded by the QR code, including unique identity ID, detailed parameters, etc., r jIt is its visual position. By connecting these positions, the motion trajectory of the object can be derived. Similarly, in this embodiment, the inertial navigation data of the robot is projected into the same coordinate system in the same way, that is, the geographical coordinate conversion of different sensors is completed. Finally, based on the geographical coordinate conversion and the pixel trajectory of the robot, the motion trajectory of the robot is obtained. Through the proposed conversion method, this embodiment can create the absolute positions of some indoor building information with simplified semantic representations in the road network, such as encoding the floor information where the camera is located, so as to complete the cross-floor robot positioning. Or establish homologous points for coordinate conversion with coordinate systems such as WGS84 to achieve seamless transition between indoor and outdoor environments.

[0064] Further, the inertial navigation sensor installed on the logistics robot measures the six degrees of freedom of the moving target's motion through a gyroscope and an accelerometer, and the six-degree-of-freedom rigid body transformation matrix obtained by integrating it is used as the main data source for inertial active positioning, which will not be elaborated here. It should be emphasized that, according to different types of robots, there are differences in inertial dead reckoning. When the target is a pedestrian or a bionic robot, inertial positioning data needs to be obtained through step frequency, step length, and direction, etc. When it is a wheeled robot, the acceleration and heading angle can be directly used. Since wheeled robots are relatively widespread, the step length model will not be discussed anymore. Since the direction change of each step of the robot is the angular velocity change relative to the original frame, and there is no elevation undulation in the indoor environment, the current moment target azimuth change can be expressed by the rotation vector on the pitch θ and yaw φ in the previous step:

[0065]

[0066] For the direction obtained at each moment, in this embodiment, the heading classification is performed through the principal component analysis (PCA) algorithm. In order to use the pattern recognition method, in this embodiment, the instantaneous direction is divided into 8 groups and the eigenvalue of each direction is learned through pre-experiments. The classified direction μ(t) can be used to match the direction result of visual positioning for position calibration. The trajectory obtained by the original inertial sensor element without fusion calibration is denoted as Traj ac :

[0067]

[0068] Traj ac is a complete trajectory record of the inertial route from start to end, and g n represents the uncalibrated inertial positioning result, and at the same time the result g n+1 at the next moment depends on the magnitudes of the horizontal acceleration and angular velocity obtained by the sensor.

[0069] In actual use, the data acquisition frequency of the camera is greater than that of the inertial data acquisition. In this embodiment, the median integration method is used to integrate the visual data at discrete times to unify the time series of the two data sources. Since the viewing angle coverage of the camera is limited, there will be blind spots in visual positioning. When the target is missed for a long time, the tracking and positioning of the target will be lost. Inertial positioning has good positioning accuracy in a short time, but the accuracy gradually increases with time. Therefore, after completing the identity matching, this embodiment uses the heading of the visual positioning result to calibrate the horizontal drift of the inertial-based positioning and constructs a visual-inertial collaborative positioning scheme. In this embodiment, a two-stage update mode is adopted to calibrate the inertial position. For the six degrees of freedom obtained by the inertial sensor, first, the visual calibration inertial attitude degree-of-freedom data is used, and the distance of the inertial within the same time window is calibrated by the distance of each frame of visual positioning SL i is expressed as:

[0070]

[0071] X t(i) and X t(i+1) represent the visual positioning results at the current and previous moments respectively. Secondly, in this embodiment, through heading calibration, the direction μ(t(i)) of each step obtained from the inertial navigation is replaced by the visual direction determined by two consecutive frames with similar time windows:

[0072]

[0073] where t(i) is a time window starting from the i-th frame. The position of the target will be recalculated and corrected according to the integral of these calibrated directions and the calibrated distance to obtain the corrected result Traj′ ac . This embodiment hopes to calibrate the inertial navigation that may have error accumulation with a more stable visual positioning result at each moment, which has not appeared in previous work.

[0074]

[0075] Secondly, the visual positioning result is further used to correct the remaining position degree-of-freedom data of the inertia. The preliminarily corrected result Traj′ ac and the visual positioning Traj pa will be used as the input of the extended Kalman filter together. The observable quantity z k is the combined matrix of the corrected values of the active and passive positioning results in the XY direction, that is, the three-axis position of the midpoint between Traj′ ac and Traj pa . H k is the measurement matrix, and v k is the measurement noise.

[0076] z k = (Traj′ ac + Traj pa ) / 2 = H k x k + v k (12)

[0077] x k = ωx k-1 + w k-1 (13)

[0078] The current state x of the pre - corrected inertial navigation k is determined jointly according to the previous state x k-1 , the process noise w k-1 , and the state transition matrix ω. The prediction part of the extended Kalman filter finally gives the predicted value of the current state quantity error and its covariance matrix P, and then uses the filtered value to feedback - correct the inertial navigation system to output the final active - passive fusion positioning result.

[0079] In one implementation, in many cases, due to the QR code being blocked, undetected, or multiple targets existing simultaneously in the scene, it will cause the failure to effectively complete the fusion task of the active - passive positioning sensors. Therefore, an active - passive trajectory matching scheme for QR code matching failure is proposed. Within a period of time, any target ob i can extract active motion features from inertial sensor devices, such as accelerometers, gyroscopes, and magnetometers. In this embodiment, sequence information including acceleration, direction, rotation, etc. is collected from them and denoted as Similarly, through the method of passive vision detection, this embodiment can also obtain the corresponding motion - related feature sequences By correlating the motion features with the trajectory data, it can effectively determine whether the trajectory to be solved belongs to the same target, and the process is as Figure 3 shown.

[0080] This embodiment uses the dynamic time warping (DTW) algorithm to recursively check all subsequence combinations of the entire sequence to obtain the best match, in order to measure the similarity score between two time series and Suppose F k(m,n) is the DTW distance between the K - th features and , which can be defined as

[0081]

[0082] where, ‖·‖ represents the Euclidean distance, which is the two - norm of the two feature sequences. This recursive definition helps this embodiment to move forward in a corresponding manner with different time sequences. The goal of DTW is to find and The mapping path between them is such that the total distance on this mapping path is minimized, and the final result is denoted as F k(M,N) 。

[0083] Then, the features are normalized, and the closer to zero indicates lower similarity:

[0084]

[0085] Among them, F Max represents obtaining the maximum distance of K features and obtaining the comprehensive similarity score between and by combining the similarity scores of K features. The passive video data is matched with the active inertial navigation data by a greedy method. For any target ob i , the best-matched active positioning device is found in this embodiment as follows:

[0086]

[0087] To avoid multiple pairings, the proposed active-passive trajectory fusion scheme does not establish a connection when the robot QR code is not scanned. When the QR code appears within the camera range, the monitoring camera can verify the pairing result by scanning the identity information and eliminate the cases of mispairing and missing pairing.

[0088] Compared with the prior art, this embodiment focuses on the robustness of complex indoor scene positioning technology. The sensors used are widely distributed and have a low threshold. It does not need to rely on high-precision mobile inertial navigation devices, nor does it need to modify the existing indoor infrastructure of many public buildings. It can realize data collection, training, and positioning tasks more conveniently and at low cost. Compared with ordinary QR code positioning methods, the TinyQR_YOLOV5 locator adapted to small-scale tasks in this embodiment can decode QR code images at different angles more quickly, accurately, and orderly, and obtain the pixel trajectory of the robot at the same time. The multi-camera perspective transformation model proposed in this embodiment can be simpler and have high accuracy, and can convert the detected pixel coordinates into real coordinates in real time to achieve passive trajectory tracking and geographical unity based on the camera. Since the multi-camera re-identification algorithm is easily affected by external conditions and causes incorrect associations, QR codes with the advantages of high tolerance, convenient and flexible use, etc. are considered. It can cover the detailed parameters of the robot and replace the original unstable association matching mechanism to achieve the association of multi-source sensors. In addition, this embodiment also considers the situation where the QR code is blocked, and further ensures the association accuracy through the feature sequences of the two trajectories. Aiming at the problem that inertial sensing is prone to cumulative errors, a fusion positioning algorithm with simple operation and high efficiency is proposed to use vision for inertial joint calibration and positioning. The method of this embodiment does not require installing instruments on the robot or using a specific sensor kit in other independent systems as a settlement point. The idea of passive vision calibration for active inertial navigation positioning has a certain degree of novelty and innovation.

[0089] Based on the above embodiment, the present invention further provides a real-time navigation system for an indoor logistics robot with enhanced passive QR code positioning, as Figure 4 shown in the figure. The system includes: a data processing module 10, a data analysis module 20, and a trajectory determination module 30. Specifically, the data processing module 10 of this embodiment is used to collect a small-scale QR code image data set, perform data enhancement processing based on a generative adversarial network to obtain an expanded data set, and divide the expanded data set into a training set and a test set. The data analysis module 20 is used to train the weights of the TinyQR_YOLOV5 model based on the training set and the test set to obtain the QR code classifier of the TinyQR_YOLOV5 model, and determine the pixel trajectory of the robot corresponding to the video data collected in real time based on the QR code classifier. The trajectory determination module 30 is used to determine the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and output the motion trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship.

[0090] The working principles of the various modules in the real-time navigation system for the indoor logistics robot with enhanced passive QR code positioning in this embodiment are the same as those of the various steps in the above method embodiment, and will not be elaborated here.

[0091] Based on the above embodiments, the present invention further provides a terminal device, and the principle block diagram of the terminal device may be as Figure 5 shown. The terminal device may include one or more processors 100 ( Figure 5 only one is shown in the figure), a memory 101, and a computer program 102 stored in the memory 101 and executable on the one or more processors 100. For example, a program for real-time navigation of an indoor logistics robot with enhanced passive QR code positioning. When the one or more processors 100 execute the computer program 102, the functions of each module / unit in the embodiment of the real-time navigation device of the indoor logistics robot with enhanced passive QR code positioning can be implemented, which is not limited herein.

[0092] In one embodiment, the processor 100 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] In one embodiment, the memory 101 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 may also include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store the computer program and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store the data that has been output or will be output.

[0094] Those skilled in the art can understand that Figure 5 the principle block diagram shown in the figure is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0095] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, operational database, or other medium used in the embodiments provided by the present invention can include non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time navigation method for an indoor logistics robot with enhanced passive positioning of two-dimensional codes, characterized in that, The method includes: Collect a small-scale QR code image dataset, perform data augmentation processing based on an adversarial generative network to obtain an augmented dataset, and divide the augmented dataset into a training set and a test set; Train the weights of the TinyQR_YOLOV5 model based on the training set and the test set to obtain a QR code classifier based on the TinyQR_YOLOV5 model, and determine the robot pixel trajectory corresponding to the video data collected in real time based on the QR code classifier; Determine the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and output the motion trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship; The determining the robot pixel trajectory corresponding to the video data collected in real time based on the QR code classifier includes: Collect the video data in real time through the cameras inherent in the scene, and input the video data into the QR code classifier in the TinyQR_YOLOV5 model to obtain the QR code border length and the central pixel coordinates respectively; Decode the image in the QR code area based on the QR code border length and the central pixel coordinates to obtain the information carried by the QR code and locate the robot pixel trajectory; The determining the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and outputting the motion trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship includes: Based on the multi-camera perspective coordinate transformation model, calculate the covariance matrix of the centralized corresponding point sets between multiple cameras, and perform singular value decomposition on the covariance matrix to determine the optimal solution; Through iterative calculations between each camera coordinate system, obtain the trajectory conversion relationship of the robot in multiple cameras; Based on the trajectory conversion relationship, project the robot inertial navigation data into the same coordinate system to obtain the geographical coordinate conversion of different sensors; Based on the geographical coordinate conversion and the robot pixel trajectory, obtain the motion trajectory of the robot.

2. The real-time navigation method of the indoor logistics robot with enhanced passive positioning of two-dimensional code according to claim 1, characterized in that The collecting a small-scale QR code image dataset, performing data augmentation processing based on an adversarial generative network to obtain an augmented dataset, and dividing the augmented dataset into a training set and a test set includes: Simulate the angles and distances of actual indoor application scenarios for shooting to collect a small-scale QR code image dataset, where the small-scale QR code image dataset contains QR code images with different indoor backgrounds, different scales, different angles, different visibilities, and different types; Process the small-scale QR code image dataset through a manual marking method to obtain the processed QR code image data and the label file corresponding to each image; Perform data augmentation processing on the processed QR code image data based on the adversarial generative network to obtain an augmented dataset.

3. The real-time navigation method of the indoor logistics robot with enhanced passive positioning of two-dimensional code according to claim 2, characterized in that, The data augmentation processing includes performing affine transformation on the QR code while keeping the background unchanged, changing the QR code type, generating partially occluded QR codes, and replacing the background while keeping the QR code unchanged.

4. The real-time navigation method of the indoor logistics robot with enhanced passive positioning of two-dimensional code according to claim 1, wherein, The TinyQR_YOLOV5 model includes a small object detection layer, which is used to constrain the predicted object range using small-sized anchors, incorporates size prior experience, deepens the upsampling process of the feature map, and performs detection after splicing the shallower feature map and the deeper feature map; Moreover, a self-attention mechanism is added to each convolutional layer in the TinyQR_YOLOV5 model.

5. The real-time navigation method for an indoor logistics robot with enhanced passive positioning of QR codes according to claim 1, characterized in that, The method further includes: Calibrating the horizontal drift based on inertial positioning according to the navigation direction of the robot's motion trajectory, and constructing a visual-inertial collaborative positioning scheme.

6. A real-time navigation system for indoor logistics robots with enhanced passive positioning of two-dimensional codes, characterized in that, The system is used to implement the steps of the real-time navigation method for an indoor logistics robot with enhanced passive QR code positioning as described in any one of claims 1-5. The system includes: A data processing module, which is used to collect a small-scale QR code image data set, perform data augmentation processing based on a generative adversarial network to obtain an augmented data set, and divide the augmented data set into a training set and a test set; A data analysis module, which is used to train the weights of the TinyQR_YOLOV5 model based on the training set and the test set to obtain a QR code classifier based on the TinyQR_YOLOV5 model, and determine the pixel trajectory of the robot corresponding to the video data collected in real time based on the QR code classifier; A trajectory determination module, which is used to determine the trajectory conversion relationship between the QR code and the robot in the multi-camera network through a multi-camera perspective coordinate transformation model, and output the motion trajectory of the robot based on the robot pixel trajectory and the trajectory conversion relationship.

7. A terminal device, characterized in that The terminal device includes a memory, a processor, and a real-time navigation program for an indoor logistics robot with enhanced passive QR code positioning stored in the memory and executable on the processor. When the processor executes the real-time navigation program for an indoor logistics robot with enhanced passive QR code positioning, the steps of the real-time navigation method for an indoor logistics robot with enhanced passive QR code positioning as described in any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a real-time navigation program for an indoor logistics robot with enhanced passive QR code positioning. When the real-time navigation program for an indoor logistics robot with enhanced passive QR code positioning is executed by a processor, the steps of the real-time navigation method for an indoor logistics robot with enhanced passive QR code positioning as described in any one of claims 1-5 are implemented.

Images