An array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items

Abstract

The present invention belongs to the technical field of multi-vehicle collaboration, and discloses an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items, which is used for synchronously positioning multiple carrier vehicles in a fleet transporting oversized items, including: a positioning module, an image identifier; the fleet includes multiple rows of vehicles distributed along a horizontal axis direction, each row of vehicles includes two carrier vehicles placed side by side, and the front of each carrier vehicle is oriented in the same direction, with the front of the carrier vehicle facing one end as the front, and the rear of the carrier vehicle facing one end as the rear; a positioning module is provided at the front end of each carrier vehicle, and the positioning module includes a camera, an IMU and a communication unit; the two carrier vehicles in the front row are used to obtain absolute posture, and the carrier vehicles in the following rows are observed by relative posture to improve the coordination between multiple vehicles, while realizing accurate real-time positioning of multiple vehicles, and improving the adaptability of the perception system in degraded environments such as tunnels and scenes with poor signals.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-vehicle collaboration, and specifically relates to an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items. Background Art

[0002] Multi-vehicle collaborative positioning is a prerequisite for improving the intelligence level of multi-vehicle collaborative transportation. During driving, due to the perception boundary limitations caused by the huge spatial size of the vehicle and the array-type transportation layout, the transport vehicle cannot directly obtain the relative position and posture of the neighboring vehicle by three-dimensional reconstruction. At the same time, there are inevitably errors in the self-positioning of the transport vehicle. If the position relationship of multiple vehicles is obtained directly based on the positioning information of a single vehicle, the positioning error of the single vehicle will be multiplied, reducing the estimation accuracy of state quantities such as stress-strain between the vehicle and the cargo, and increasing the risk of damage to oversized items. On the other hand, when transport vehicles are driving in areas with poor signals such as tunnels and structurally degraded environments such as wilderness, solutions based on vehicle-mounted RTK or lidar may have problems with algorithm performance and reliability degradation or even disability; therefore, in the prior art, there is a problem of insufficient coordination between multiple vehicles when multiple vehicles transport oversized items. Summary of the invention

[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide an array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items, which solves the problem in the prior art of insufficient coordination between multiple vehicles when multiple vehicles transport oversized items.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] An array-type multi-vehicle collaborative visual positioning system for transporting oversized items is used to synchronously position multiple transport vehicles in a fleet transporting oversized items, including: a positioning module, an image identifier;

[0006] The vehicle fleet includes multiple rows of vehicles distributed along the horizontal axis, each row of vehicles includes two transport vehicles placed side by side, and the front of each transport vehicle faces the same direction, with the front of the transport vehicle facing one end as the front, and the rear of the transport vehicle facing one end as the rear;

[0007] A positioning module is installed at the front end of each carrier vehicle, and the positioning module includes a camera, an IMU and a communication unit;

[0008] The positioning modules on the two vehicles in the front row are respectively set as a pilot positioning module and an absolute positioning module;

[0009] The pilot positioning module also includes a fusion calculation unit;

[0010] The absolute positioning module also includes a positioning calculation unit;

[0011] The rear end of each transport vehicle in front of the last row of transport vehicles is provided with an image mark;

[0012] The positioning modules on each transport vehicle behind the front row of transport vehicles are all set as relative positioning modules, and the relative positioning modules also include a control unit and a lighting module for illuminating the image identification. The number of cameras in any relative positioning module is set to two, and the two cameras are respectively used to capture the two image identifications on the two transport vehicles in the front row.

[0013] A method for transporting oversized items using an array-type multi-vehicle collaborative visual positioning system for transporting oversized items is disclosed, which uses an array-type multi-vehicle collaborative visual positioning system for transporting oversized items to synchronously position multiple transport vehicles in a fleet transporting oversized items, comprising the following steps:

[0014] Initialization of array multi-vehicle positioning system;

[0015] Absolute position observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module;

[0016] Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module;

[0017] Collaborative positioning is performed by transmitting various information to the pilot positioning module and combining it with the spatial constraints of oversized parts.

[0018] The array multi-vehicle positioning system initialization specifically includes the following steps:

[0019] Based on the coordinate systems of each IMU, establish the coordinate systems of each positioning module;

[0020] RTK is temporarily arranged at the connection points between each carrier and the oversized piece, and the vehicle coordinate system of each carrier is established based on each connection point;

[0021] Each carrier drives the corresponding positioning module;

[0022] Get the camera coordinate system of each positioning module respectively Displacement observation point under ,in , and at the same time obtain the coordinate system of each IMU in its corresponding positioning module Displacement observation point under , the rigid transformation matrix from the coordinates in the positioning module coordinate system to the coordinates in the camera coordinate system is defined as , The expression is as follows:

[0023]

[0024] Obtain the displacement observation points of the positioning module in the positioning module coordinate system respectively and RTK displacement observation points in vehicle coordinate system ,in , define the vehicle coordinate system The rigid transformation matrix from the central coordinate to the central coordinate of the positioning module coordinate system is: , The expression is as follows:

[0025]

[0026] The two cameras in the relative positioning module respectively capture the two image marks on the two transport vehicles in the front row, and the position of each image mark in the camera coordinate system of the camera that captured it is obtained through the Perspective-n-Point algorithm. ;

[0027] Assume that the vehicle coordinate system of any vehicle with image identification is , let the vehicle coordinate system of any of the two vehicles in the row after the image mark be , and define Central coordinates to The rigid transformation matrix of the mid-coordinate is , then it can be calculated that the image mark is in the vehicle coordinate system of the carrier vehicle where the image mark is set The posture in , The expression is as follows:

[0028]

[0029] When oversized items are installed on a transport vehicle, any vehicle in the rear row of any two adjacent rows of vehicles Capture any vehicle in the front row The image ID is obtained in the carrier vehicle. The pose of the camera used to capture the image mark in the camera coordinate system is , Carrier The coordinates in the vehicle coordinate system are the same as those of the carrier vehicle The rotation matrix between the coordinates in the vehicle coordinate system is set to ,in , The expression is as follows:

[0030]

[0031] in, , , and They are all calibration external parameters of the array multi-vehicle positioning system.

[0032] The absolute position and posture observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module. The specific steps are as follows:

[0033] Based on the cameras in the navigation positioning module and the absolute positioning module, the coordinates of the environmental landmarks in their respective camera coordinate systems are obtained. ,in ;

[0034] By calibrating the external parameters for conversion, the coordinates of the environmental landmarks in the respective vehicle coordinate systems of the pilot positioning module and the absolute positioning module are obtained. ,in ;

[0035] Measured by the IMUs in the pilot positioning module and the absolute positioning module;

[0036] The absolute positioning module obtained And the IMU measurement value Transmitted to the pilot positioning module, ,in Represents the acceleration measured by the IMU in the vehicle coordinate system, Represents the angular velocity measured by the IMU in the vehicle coordinate system.

[0037] Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module. The specific steps are as follows:

[0038] The lighting module in the relative positioning module illuminates the front image mark;

[0039] The corresponding image mark is photographed by the camera in each relative positioning module, and the spatial information of the image mark is obtained ;

[0040] Spatial information of the image mark Convert to the vehicle coordinate system of the vehicle on which the image mark is installed, and combine with the calibration external parameters , obtain the rigidity change matrix between the vehicle coordinate system of the carrier vehicle installed by the current relative positioning module and the vehicle coordinate system of the carrier vehicle installed by the captured image mark , and at the same time obtain the IMU measurement value in the vehicle coordinate system of the carrier vehicle where the relative positioning module is installed ;

[0041] The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation positioning module.

[0042] Through the information transmitted to the pilot positioning module and combined with the spatial constraints of the oversized parts, collaborative positioning is performed. The specific steps are as follows:

[0043] Assume the number of relative positioning modules is n ;

[0044] The state change of the array multi-vehicle positioning system is defined as follows:

[0045]

[0046] in, , ,in ... n ;

[0047] When initializing the array multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle installed by the pilot positioning module is the global coordinate system , , Respectively represent The position of the vehicle equipped with the relative positioning module and the first row of vehicles in the global coordinate system at all times;

[0048] represent The state of the IMU in the relative positioning module at the moment in the vehicle coordinate system of the carrier vehicle on which the relative positioning module is installed, including the obtained speed, angular velocity and its own zero bias;

[0049] Through the pre-integration method, IMU Time to The estimation of the position change of the vehicle at each moment is as follows:

[0050]

[0051] in, represent i The rotation matrix, translation vector, velocity vector, gyroscope and accelerometer bias at the moment, represents the sampling time interval of IMU, ~ represents the observed value, assuming zero bias , Satisfy the Wiener process, measure the noise , is the discretized zero-mean Gaussian white noise, Represents a state quantity from i Time has come j The amount of change at a moment;

[0052] Based on the IMU measurement value, construct the IMU measurement value residual:

[0053]

[0054] The transport vehicle is equipped with a vehicle-cargo connection support platform, and the oversized cargo support is placed on the vehicle-cargo connection support platform. The relationship between the vehicle-cargo connection support platform of the transport vehicle and the oversized cargo can be simplified to a hinged relationship, so the displacement difference between the front and rear transport vehicles in any two adjacent rows of vehicles can be obtained. ,in , build cargo residuals based on the size of the oversized piece as follows:

[0055]

[0056] Based on the coordinates of the environmental landmarks in the vehicle coordinate system of the pilot positioning module and the absolute positioning module The changes construct the landmark residuals as follows:

[0057]

[0058] The relative position between the vehicles obtained by observing the image mark based on the relative positioning module , construct the relative pose observation residual as follows:

[0059]

[0060] Comprehensively consider the constructed residuals, use sliding window algorithm and factor graph optimization to solve the optimal , and then coordinate the positioning of multiple vehicles.

[0061] The sliding window algorithm and factor graph optimization are used to solve the optimal , the specific steps are as follows:

[0062] Define the prior factor as the prior constraint of the key frame in the sliding window algorithm. After each update optimization, use the marginalization method to obtain the covariance of the next key frame pose and use it as the prior factor for the next round of optimization to construct the prior factor marginalization residual ;

[0063] Find the best solution The calculation formula is as follows:

[0064]

[0065] in, represents the Huber norm, represents the Mahalanobis distance, Represents the IMU measurement residual, represents the landmark residual, Represents the residual of goods, represents the marginalized residual of the prior factor, Represents the relative pose observation residual.

[0066] Beneficial effects of the present invention:

[0067] An array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items provided in an embodiment of the present invention adopts a multi-source perception method such as the two transport vehicles in the front row obtain the absolute position and posture, and the transport vehicles in the following rows all observe the relative position and posture to realize array-type multi-vehicle collaborative positioning for oversized collaborative transportation. While realizing accurate real-time positioning of multiple vehicles, it improves the adaptability of the array-type multi-vehicle positioning system in degraded environments such as tunnels and signal-poor scenarios, and also improves the flexibility of sensor layout when facing spatial constraints of irregular oversized cargo. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 is a flow chart of the array type multi-vehicle positioning method of the present invention;

[0070] Figure 2 It is a schematic diagram of the positions of the pilot positioning module, the absolute positioning module and the relative positioning module of the present invention;

[0071] Figure 3 It is a schematic diagram of the position relationship between the image identification and relative positioning modules of the present invention;

[0072] Figure 4 It is a schematic diagram of the rigid transformation matrix between the vehicle coordinate systems of any two adjacent vehicles in front and behind of the present invention;

[0073] Figure 5 It is a schematic diagram of the principle of multi-vehicle collaborative positioning of the sliding window algorithm of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] like Figures 1 to 5 As shown, an array-type multi-vehicle collaborative visual positioning system for transporting oversized items is used to synchronously position multiple transport vehicles in a fleet transporting oversized items, including: a positioning module, an image identifier;

[0076] The vehicle fleet includes multiple rows of vehicles distributed along the horizontal axis, each row of vehicles includes two transport vehicles placed side by side, and the front of each transport vehicle faces the same direction, with the front of the transport vehicle facing one end as the front, and the rear of the transport vehicle facing one end as the rear;

[0077] A positioning module is installed at the front end of each carrier vehicle, and the positioning module includes a camera, an IMU and a communication unit;

[0078] The positioning modules on the two vehicles in the front row are respectively set as a pilot positioning module and an absolute positioning module;

[0079] The pilot positioning module also includes a fusion calculation unit;

[0080] The absolute positioning module also includes a positioning calculation unit;

[0081] The rear end of each transport vehicle in front of the last row of transport vehicles is provided with an image mark;

[0082] The positioning modules on each transport vehicle behind the front row of transport vehicles are all set as relative positioning modules, and the relative positioning modules also include a control unit and a lighting module for illuminating the image identification. The number of cameras in any relative positioning module is set to two, and the two cameras are respectively used to capture the two image identifications on the two transport vehicles in the front row.

[0083] Preferably, the fusion computing unit may be an NVIDIA Jetson Orin NX fusion computing unit;

[0084] Preferably, the positioning computing unit may use an NVIDIA Jetson Orin Nano positioning computing unit;

[0085] Preferably, any communication unit supports at least one or more of Bluetooth, Wi-Fi, ZigBee network and Lora communication, uses TCP protocol to communicate with other communication units, and ensures reliable data transmission;

[0086] Preferably, the control unit can be a Raspberry Pi 5B control unit;

[0087] Preferably, the cameras in the pilot positioning module and the absolute positioning module can use Intel D435i depth cameras to further improve positioning accuracy;

[0088] Preferably, the cameras in the relative positioning module are all monocular cameras, and a 1080P 30 frame industrial camera can be used, and the lens is a 2.02mm focal length 130° field angle non-distortion lens, which can reduce the overall cost of the system while ensuring effective shooting and capturing of image identification;

[0089] Preferably, the image marker uses a glass-based material, and the image marker includes but is not limited to characteristic markers such as ArUco and AprilTag. Figure 3 As shown, each image mark is different from the image marks on at least two vehicles in the same row, and the two image marks in the same row have different meanings, such as representing 1 and 2, so as to avoid the problem of repeated or wrong recognition of image marks and improve the effectiveness and accuracy of the image mark pose estimation;

[0090] Preferably, the lighting module may use infrared LEDs to reduce the impact of ambient light changes on image identification, and its perception range is greater than that of the camera.

[0091] like Figure 1 As shown, an array-type multi-vehicle collaborative visual positioning method for transporting oversized items uses an array-type multi-vehicle collaborative visual positioning system for transporting oversized items to synchronously position multiple transport vehicles in a fleet of oversized items, including the following steps:

[0092] Initialization of array multi-vehicle positioning system;

[0093] Absolute position observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module;

[0094] Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module;

[0095] Collaborative positioning is performed by transmitting various information to the navigation positioning module and combining it with the spatial constraints of oversized items.

[0096] The array multi-vehicle positioning system initialization specifically includes the following steps:

[0097] Based on the coordinate systems of each IMU, establish the coordinate systems of each positioning module;

[0098] RTK is temporarily arranged at the connection points between each carrier and the oversized piece, and the vehicle coordinate system of each carrier is established based on each connection point;

[0099] Each carrier drives the corresponding positioning module;

[0100] Get the camera coordinate system of each positioning module respectively Displacement observation point under ,in , and at the same time obtain the coordinate system of each IMU in its corresponding positioning module Displacement observation point under , the rigid transformation matrix from the coordinates in the positioning module coordinate system to the coordinates in the camera coordinate system is defined as , The expression is as follows:

[0101]

[0102] Among them, the rigid transformation matrix T By the rotation matrix and displacement vector For the convenience of calculation, in the subsequent calculation process, the rotation matrix R will be transformed into a rotation vector through the logarithmic mapping in the three-dimensional rotation group SO(3), where ;

[0103]

[0104] in, Represents a mapping from an antisymmetric matrix to a vector.

[0105] Obtain the displacement observation points of the positioning module in the positioning module coordinate system respectively and RTK displacement observation points in vehicle coordinate system ,in , define the vehicle coordinate system The rigid transformation matrix from the central coordinate to the central coordinate of the positioning module coordinate system is: , The expression is as follows:

[0106]

[0107] The two cameras in the relative positioning module respectively capture the two image marks on the two transport vehicles in the front row, and the position of each image mark in the camera coordinate system of the camera that captured it is obtained through the Perspective-n-Point algorithm. ;

[0108] Assume that the vehicle coordinate system of any vehicle with image identification is , let the vehicle coordinate system of any of the two vehicles in the row after the image mark be , and define Central coordinates to The rigid transformation matrix of the mid-coordinate is , then it can be calculated that the image mark is in the vehicle coordinate system of the carrier vehicle where the image mark is set Position in , The expression is as follows:

[0109]

[0110] like Figure 4 As shown in the figure, when the oversized items are installed on the transport vehicle, any of the two adjacent rows of transport vehicles in the rear row Capture any vehicle in the front row The image ID is obtained in the carrier vehicle. The pose of the camera used to capture the image mark in the camera coordinate system is , Carrier The coordinates in the vehicle coordinate system are the same as those of the carrier vehicle The rotation matrix between the coordinates in the vehicle coordinate system is set to ,in , The expression is as follows:

[0111]

[0112] in, , , and They are all calibration external parameters of the array multi-vehicle positioning system.

[0113] The absolute position and posture observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module. The specific steps are as follows:

[0114] Based on the cameras in the navigation positioning module and the absolute positioning module, the coordinates of the environmental landmarks in their respective camera coordinate systems are obtained. ,in ;

[0115] By calibrating the external parameters for conversion, the coordinates of the environmental landmarks in the respective vehicle coordinate systems of the pilot positioning module and the absolute positioning module are obtained. ,in ;

[0116] Measured by the IMUs in the pilot positioning module and the absolute positioning module;

[0117] The absolute positioning module obtained And the IMU measurement value Transmitted to the pilot positioning module, ,in Represents the acceleration measured by the IMU in the vehicle coordinate system, Represents the angular velocity measured by the IMU in the vehicle coordinate system.

[0118] Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module. The specific steps are as follows:

[0119] The lighting module in the relative positioning module illuminates the front image mark;

[0120] The corresponding image mark is photographed by the camera in each relative positioning module, and the spatial information of the image mark is obtained ;

[0121] Spatial information of the image mark Convert to the vehicle coordinate system of the vehicle on which the image mark is installed, and combine with the calibration external parameters , obtain the rigidity change matrix between the vehicle coordinate system of the carrier vehicle installed by the current relative positioning module and the vehicle coordinate system of the carrier vehicle installed by the captured image mark ;

[0122] The above conversion process is: , , and then solve for ;

[0123] At the same time, the IMU measurement value in the vehicle coordinate system of the carrier vehicle on which the relative positioning module is installed is obtained ;

[0124] The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation positioning module.

[0125] Through the information transmitted to the pilot positioning module and combined with the spatial constraints of the oversized parts, collaborative positioning is performed. The specific steps are as follows:

[0126] Assume the number of relative positioning modules is n ;

[0127] The state change of the array multi-vehicle positioning system is defined as follows:

[0128]

[0129] in, , ,in ... n ;

[0130] When initializing the array multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle installed by the pilot positioning module is the global coordinate system , , Respectively represent The position of the vehicle equipped with the relative positioning module and the first row of vehicles in the global coordinate system at all times;

[0131] represent The state of the IMU in the relative positioning module at the moment in the vehicle coordinate system of the carrier vehicle on which the relative positioning module is installed, including the obtained speed, angular velocity and its own zero bias;

[0132] Through the pre-integration method, IMU Time to The estimation of the vehicle posture change at each moment is as follows: represent i The rotation matrix, translation vector, velocity vector, gyroscope and accelerometer bias at the moment, represents the sampling time interval of IMU, ~ represents the observed value, assuming zero bias , Satisfy the Wiener process, measure the noise , is the discretized zero-mean Gaussian white noise, Represents a state quantity from i Time has come j The amount of change at a moment;

[0133]

[0134] Based on the IMU measurement value, construct the IMU measurement value residual ( Figure 5 IMU residual in short form):

[0135]

[0136] in, is a composite function of the logarithmic mapping and the mapping of antisymmetric matrices to vectors in the three-dimensional rotation group SO(3), transforming the rotation matrix becomes a rotation vector;

[0137] like Figure 2 As shown in the figure, the oversized cargo is regarded as a rigid body model. The carrier is equipped with a vehicle-cargo connection support platform, and the oversized cargo support is placed on the vehicle-cargo connection support platform. Therefore, the relationship between the vehicle-cargo connection support platform of the carrier and the oversized cargo can be simplified to a hinged relationship. Then, the displacement difference between the front carrier and the rear carrier in any two adjacent rows of vehicles can be obtained. ,in , build cargo residuals based on the size of the oversized piece as follows:

[0138]

[0139] Based on the coordinates of the environmental landmarks in the vehicle coordinate system of the pilot positioning module and the absolute positioning module The changes construct the landmark residuals as follows:

[0140]

[0141] The relative position between the vehicles obtained by observing the image mark based on the relative positioning module , construct the relative pose observation residual ( Figure 5 ), as follows:

[0142]

[0143] Comprehensively consider the constructed residuals, use sliding window algorithm and factor graph optimization to solve the optimal , and then coordinate the positioning of multiple vehicles.

[0144] like Figure 5 As shown, the sliding window algorithm and factor graph optimization are used to solve the optimal , the specific steps are as follows:

[0145] Define the prior factor as the prior constraint of the key frame in the sliding window algorithm. After each update optimization, use the marginalization method to obtain the covariance of the next key frame pose and use it as the prior factor for the next round of optimization to construct the prior factor marginalization residual ( Figure 5 a priori residual in short form);

[0146] Find the best solution The calculation formula is as follows:

[0147]

[0148] in, represents the Huber norm, represents the Mahalanobis distance, Represents the IMU measurement residual, represents the landmark residual, Represents the residual of goods, represents the marginalized residual of the prior factor, Represents the relative pose observation residual. In the sliding window algorithm, each vehicle takes 3 key frame poses.

[0149] An array-type multi-vehicle collaborative visual positioning system and method for transporting oversized items provided in an embodiment of the present invention adopts a multi-source perception method such as the two transport vehicles in the front row obtain the absolute position and posture, and the transport vehicles in the following rows all observe the relative position and posture to realize array-type multi-vehicle collaborative positioning for oversized collaborative transportation. While realizing accurate real-time positioning of multiple vehicles, it improves the adaptability of the array-type multi-vehicle positioning system in degraded environments such as tunnels and signal-poor scenarios, and also improves the flexibility of sensor layout when facing spatial constraints of irregular oversized cargo.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. An array-type multi-vehicle collaborative visual positioning system for transporting oversized items, used to synchronously position multiple transport vehicles in a fleet transporting oversized items, characterized in that: include: Positioning module, image identification; The vehicle fleet includes multiple rows of vehicles distributed along the horizontal axis, each row of vehicles includes two transport vehicles placed side by side, and the front of each transport vehicle faces the same direction, with the front of the transport vehicle facing one end as the front, and the rear of the transport vehicle facing one end as the rear; A positioning module is installed at the front end of each carrier vehicle, and the positioning module includes a camera, an IMU and a communication unit; The positioning modules on the two vehicles in the front row are respectively set as a pilot positioning module and an absolute positioning module; The pilot positioning module also includes a fusion computing unit, which is an NVIDIA Jetson Orin NX fusion computing unit; The absolute positioning module also includes a positioning calculation unit, which is an NVIDIA Jetson Orin Nano positioning calculation unit; The rear end of each transport vehicle in front of the last row of transport vehicles is provided with an image mark; The positioning modules on each transport vehicle behind the front row of transport vehicles are all set as relative positioning modules, and the relative positioning modules also include a control unit and a lighting module for illuminating the image identification. The number of cameras in any relative positioning module is set to two, and the two cameras are respectively used to capture the two image identifications on the two transport vehicles in the front row.

2. A method for coordinated visual positioning of multiple vehicles in an array for transporting oversized items, which uses the coordinated visual positioning system for transporting oversized items in an array as described in claim 1 to synchronously position multiple vehicles in a fleet for transporting oversized items, characterized in that: The following steps are involved: Initialization of array multi-vehicle positioning system; Absolute position observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module; Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module; Through the information transmitted to the pilot positioning module, combined with the spatial constraints of the oversized parts, collaborative positioning is performed; Through the information transmitted to the pilot positioning module and combined with the spatial constraints of the oversized parts, collaborative positioning is performed. The specific steps are as follows: Assume the number of relative positioning modules is n ; Defining the state change of the array multi-vehicle positioning system as follows: in, , ,in ... n ; When initializing the array multi-vehicle positioning system, the vehicle coordinate system of the carrier vehicle installed by the pilot positioning module is the global coordinate system , , Respectively represent The position of the vehicle equipped with the relative positioning module and the first row of vehicles in the global coordinate system at all times; represent The state of the IMU in the relative positioning module at the moment in the vehicle coordinate system of the carrier vehicle on which the relative positioning module is installed, including the obtained speed, angular velocity and its own zero bias; Through the pre-integration method, IMU Time to The estimation of the position change of the vehicle at each moment is as follows: in, represent i The rotation matrix, translation vector, velocity vector, gyroscope and accelerometer bias at the moment, represents the sampling time interval of IMU, ~ represents the observed value, assuming zero bias , Satisfy the Wiener process, measure the noise , is the discretized zero-mean Gaussian white noise, Represents a state quantity from i Time has come j The amount of change at a moment; Based on the IMU measurement value, construct the IMU measurement value residual: The transport vehicle is equipped with a vehicle-cargo connection support platform, and the oversized cargo support is placed on the vehicle-cargo connection support platform. The relationship between the vehicle-cargo connection support platform of the transport vehicle and the oversized cargo can be simplified to a hinged relationship, so the displacement difference between the front and rear transport vehicles in any two adjacent rows of vehicles can be obtained. ,in , build cargo residuals based on the size of the oversized piece as follows: Based on the coordinates of the environmental landmarks in the vehicle coordinate system of the pilot positioning module and the absolute positioning module The changes construct the landmark residuals as follows: The relative position between the vehicles obtained by observing the image mark based on the relative positioning module , construct the relative pose observation residual as follows: Comprehensively consider the constructed residuals, use sliding window algorithm and factor graph optimization to solve the optimal , and then coordinate the positioning of multiple vehicles; The sliding window algorithm and factor graph optimization are used to solve the optimal , the specific steps are as follows: Define the prior factor as the prior constraint of the key frame in the sliding window algorithm. After each update optimization, use the marginalization method to obtain the covariance of the next key frame pose and use it as the prior factor for the next round of optimization to construct the prior factor marginalization residual ; Find the best solution The calculation formula is as follows: in, represents the Huber norm, represents the Mahalanobis distance, Represents the IMU measurement residual, represents the landmark residual, Represents the residual of goods, represents the marginalized residual of the prior factor, Represents the relative pose observation residual.

3. The array-type multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 2 is characterized in that: The array multi-vehicle positioning system initialization specifically includes the following steps: Based on each IMU coordinate system, establish the positioning module coordinate system of each positioning module; RTK is temporarily arranged at the connection points between each carrier and the oversized piece, and the vehicle coordinate system of each carrier is established based on each connection point; Each carrier drives the corresponding positioning module; Get the camera coordinate system of each positioning module respectively Displacement observation point under ,in , and at the same time obtain the coordinate system of each IMU in its corresponding positioning module Displacement observation point under , the rigid transformation matrix from the coordinates in the camera coordinate system to the coordinates in the positioning module coordinate system is defined as , The expression is as follows: Obtain the displacement observation points of the positioning module in the positioning module coordinate system respectively and RTK displacement observation points in vehicle coordinate system ,in , define the coordinates in the positioning module coordinate system to the vehicle coordinate system The rigid transformation matrix of the mid-coordinate is , The expression is as follows: The two cameras in the relative positioning module respectively capture the two image marks on the two transport vehicles in the front row, and the position of each image mark in the camera coordinate system of the camera that captured it is obtained through the Perspective-n-Point algorithm. ; Assume that the vehicle coordinate system of any vehicle with image identification is , let the vehicle coordinate system of any of the two vehicles in the row after the image mark be , and define Central coordinates to The rigid transformation matrix of the mid-coordinate is , then it can be calculated that the image mark is in the vehicle coordinate system of the carrier vehicle where the image mark is set Position in , The expression is as follows: When oversized items are installed on a transport vehicle, any vehicle in the rear row of any two adjacent rows of vehicles Capture any vehicle in the front row The image ID is obtained in the carrier vehicle. The pose of the camera used to capture the image mark in the camera coordinate system is , Carrier The coordinates in the vehicle coordinate system are the same as those of the carrier vehicle The rotation matrix between the coordinates in the vehicle coordinate system is set to ,in , The expression is as follows: in, , , and They are all calibration external parameters of the array multi-vehicle positioning system.

4. The array-type multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 3 is characterized in that: The absolute position and posture observation is performed through the pilot positioning module and the absolute positioning module, and the information obtained by the absolute positioning module is transmitted to the pilot positioning module. The specific steps are as follows: Based on the cameras in the navigation positioning module and the absolute positioning module, the coordinates of the environmental landmarks in their respective camera coordinate systems are obtained. ,in ; By calibrating the external parameters for conversion, the coordinates of the environmental landmarks in the respective vehicle coordinate systems of the pilot positioning module and the absolute positioning module are obtained. ,in ; Measured by the IMUs in the pilot positioning module and the absolute positioning module; The absolute positioning module obtained And the IMU measurement value Transmitted to the pilot positioning module, ,in Represents the acceleration measured by the IMU in the vehicle coordinate system, Represents the angular velocity measured by the IMU in the vehicle coordinate system.

5. The array-type multi-vehicle collaborative visual positioning method for transporting oversized items according to claim 4 is characterized in that: Each relative positioning module performs relative posture observation and transmits the information obtained by each relative positioning module to the pilot positioning module. The specific steps are as follows: The lighting module in the relative positioning module illuminates the front image mark; The corresponding image mark is photographed by the camera in each relative positioning module, and the spatial information of the image mark is obtained ; Spatial information of the image mark Convert to the vehicle coordinate system of the vehicle on which the image mark is installed, and combine with the calibration external parameters , obtain the rigidity change matrix between the coordinates in the vehicle coordinate system of the carrier vehicle installed by the current relative positioning module and the coordinates in the vehicle coordinate system of the carrier vehicle installed by the captured image mark , and at the same time obtain the IMU measurement value in the vehicle coordinate system of the carrier vehicle where the relative positioning module is installed ; The rigidity change matrix obtained by each relative positioning module and IMU measurements Transmitted to the navigation positioning module.

Images