A multi-sensor fusion multi-terminal target recognition and positioning method

Abstract

The present invention relates to the field of image processing technology, and in particular to a multi-terminal target recognition and positioning method of multi-sensor fusion, including stack preprocessing of camera data and lidar data; time synchronization and anti-jitter; building a neural network for target detection; image and radar information fusion and joint search, fusing point clouds with images and searching for objects on the image; and performing spatial position calculation. The present invention is based on fixed navigation of robots in fixed scenes with a fixed perspective, and has the characteristics of high cohesion and low coupling. This automatic driving system available in all scenarios can realize a set of systems and multi-terminal use, greatly reducing the equipment cost and the cost of modifying existing equipment. It has a more comprehensive overall consideration of all scenarios, and the coordinated scheduling between multiple targets is more stable and efficient. It can be used in scenes such as warehouses, transportation, or closed parks. With the help of one or more high-position perspectives, multi-target and large-scale guidance can be performed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-sensor fusion multi-terminal target recognition and positioning method. Background Art

[0002] In recent years, with the development of computer vision, autonomous driving systems have become an increasingly popular field. Currently, the industry's solutions for autonomous driving applications mostly use laser guidance, RFID guidance, magnetic strip guidance, and vehicle-side SLAM mapping robots.

[0003] Laser guidance is mainly achieved by the reflection of light. Precisely positioned laser reflectors are installed around the vehicle's travel path. Trigonometric calculations are used to determine the vehicle's current position and direction to guide the vehicle.

[0004] RFID guidance technology is the most widely used technology. It buries metal wires along the vehicle's travel path and loads low-frequency, low-voltage current to generate a magnetic field around the wires. The induction coil on the vehicle guides the vehicle by identifying and tracking the strength of the navigation magnetic field.

[0005] Magnetic guidance involves placing a magnetic navigation sensor on the vehicle, positioned directly above the magnetic stripe. When the vehicle's trajectory aligns with the guidance path, the magnetic guidance sensor is positioned directly above the magnetic stripe marker. The sensor detects zero path deviation, and the controller does not perform any deviation correction, allowing the vehicle to maintain its original trajectory.

[0006] Vehicle-side SLAM mapping refers to installing a camera on the vehicle, collecting image information through visual sensors while the vehicle is driving, and determining the vehicle's current position by processing the image information.

[0007] The problem with the first three is that they are highly dependent on pre-set maps and various identification sensors, which imposes great restrictions on applications with a high degree of freedom.

[0008] The vehicle-based SLAM makes up for the shortcomings of the above three and has a greater degree of freedom, but each of its devices requires a high-precision sensor to realize the perception of the environment. This increases the cost of a single device and it is not easy to synchronize information and schedule tasks between devices. Summary of the Invention

[0009] In response to the shortcomings of existing algorithms, the present invention is based on fixed robot navigation in fixed scenes with a fixed perspective, which has the characteristics of high cohesion and low coupling. This autonomous driving system that can be used in all scenarios can realize a set of systems and be used on multiple terminals, greatly reducing the equipment cost and the cost of modifying existing equipment. It has a more comprehensive overall consideration of all scenarios, and the coordinated scheduling between multiple targets is more stable and efficient. It can be used in scenarios such as warehouses, transportation or closed parks, and multi-target and large-scale guidance can be carried out with the help of one or more high-position perspectives.

[0010] The technical solution adopted by the present invention is: a multi-sensor fusion multi-terminal target recognition and positioning method includes the following steps:

[0011] Step 1: Preprocess the camera data and lidar data by stacking;

[0012] Furthermore, the stack obtains sensor data through a receiving loop, which is managed by a receiving thread running in a special loop. First, the items in the short-term stack are stacked and the corresponding results are generated; secondly, the minimum long-term stack containing the items in the short-term stack is retrieved, and the required result stack is generated using the results of the short-term stack and the remaining item data; finally, all stack operations are terminated and all stack results are sent out.

[0013] It is worth mentioning that the timing of starting each stack is when new sensor data is fed in, so that it can be updated at a higher frequency

[0014] Step 2: Time synchronization and jitter prevention;

[0015] Further, it specifically includes:

[0016] Use a structure to encapsulate data and timestamps and send them to the queue. At the same time, use timestamps for time synchronization when processing data. By detecting the difference between the timestamps of point cloud data and image data, if it is greater than a certain threshold, discard the data on the older side and wait for new data to be transmitted from that side. If the requirements are still not met after waiting for the specified number of rounds, all data of the thread will be discarded and new data will be obtained.

[0017] Step 3: Build a neural network for target detection;

[0018] Further, it specifically includes:

[0019] The first layer uses the PicoDet network to identify the vehicle body and positioning points, extracts the ROI from the vehicle body and performs image cropping; the image is sent to the second layer's Yolov3 network for position recognition of the landmark points; the third layer uses the CNN network for one-stop recognition and classification of the ROI by color and digital labeling.

[0020] Furthermore, the third layer also includes screening through brightness threshold and HSV threshold to realize the recognition of the color of the marking point.

[0021] Step 4: Image and radar information fusion and joint search: fuse the point cloud with the image and search for objects on the image;

[0022] Further, it specifically includes:

[0023] S41, obtaining a rotation matrix and a translation matrix through joint calibration, and transforming each point in the 3D point cloud into a point in the camera coordinate system, and representing the transformation process with a transformation matrix;

[0024] S42, correcting the original coordinates of the camera to the actual imaging coordinates of the camera;

[0025] S43. First, project the 3D point cloud to obtain a mapping table of all spatial points to the image plane, search and extract points within the actual image range, and delete those outside the camera image range;

[0026] Secondly, the neural network searches for a specific target and selects a set of points in the point cloud that may constitute the target;

[0027] S44. Use the cluster averaging method to convert the point set constituting the target object into an approximate point in space, and encapsulate it into a search function. The search function accepts a pixel coordinate range and expands it using the expansion amount to obtain an aggregated world coordinate.

[0028] Step 5: Perform spatial position calculation;

[0029] Further, it specifically includes:

[0030] S51. Obtain the coordinates of two points X1 = (x1, y1, z1) and X2 = (x2, y2, z2) in the radar point cloud coordinates through a search function, and construct a spatial vector pointing from the origin to the two coordinates.

[0031] S52, unify the radar and vehicle coordinate systems in the direction and establish a polar coordinate system on the horizontal plane;

[0032] S53, constructing the spatial vectors of the near and far positioning points; constructing the spatial vector from the vehicle to the target point, and calculating the distance between the vehicle and the target;

[0033] S54, obtaining the direction angle by using the angle formula between the two space vectors in step S53;

[0034] S55, converting the polar coordinate information of the target point in the polar coordinate system to a point on the XY plane in the vehicle coordinate system;

[0035] S56, obtaining the position of the target in the polar coordinate system;

[0036] S57. Use SLAM to estimate the position and fuse the IMU to obtain the posture conversion information from the polar coordinate point to the vehicle coordinate system and then to the SLAM mapping coordinate system.

[0037] Step 6: Use visual SLAM to build maps and navigate to achieve robot obstacle avoidance and path planning.

[0038] Beneficial effects of the present invention:

[0039] 1. The use of the stack method can increase the frequency of information updates while ensuring the amount of information obtained.

[0040] 2. The position solution method uses space vectors and polar coordinates, eliminating the need to find multiple rigorous spatial positioning points and making it easier to transform and specify coordinates. The polar coordinate system with the vehicle as the origin can describe the position more simply and does not rely much on a fixed absolute coordinate.

[0041] 3. The car moves forward while finding its way. There is no need to worry about obstructions between the car and the target. The car's built-in SLAM function is mainly used for local navigation and obstacle avoidance, and there is no rigid requirement for whether the target is on the map. At the same time, for targets already on the map, the car can use its own SLAM map as a cache to optimize pathfinding speed;

[0042] 4. By setting up high-precision sensors (lidar + vision) with fixed viewing angles, the vehicle side and the sensor can be decoupled while obtaining high-precision real-time perception of the environment, allowing the addition of robots at a lower marginal cost.

[0043] 5. In addition, this unified backend-based perception system has a more comprehensive understanding of the overall environment and behavior prediction, and can also more efficiently coordinate robot path planning;

[0044] 6. Due to the adoption of a unified perception system, each set of sensor modules is consistent, thus reducing the process of debugging sensor parameters;

[0045] 7. Build a real-time slam map based on a real-time updated environment with high flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the radar base station end and vehicle end system operation diagram of the present invention;

[0047] Figure 2 is a schematic diagram of the laser radar and camera hardware of the present invention;

[0048] Figure 3 It is a logic diagram of the connection between the sensor and the computing platform of the present invention;

[0049] Figure 4 It is a multi-terminal target recognition and positioning method of multi-sensor fusion of the present invention;

[0050] Figure 5 It is a schematic diagram of the stacking process of the present invention;

[0051] Figure 6 It is the neural network operation process of the present invention;

[0052] Figure 7 This is a neural network recognition effect diagram of the present invention;

[0053] Figure 8 is a flow chart of the fusion and joint search of the present invention;

[0054] Figure 9 It is a schematic diagram of the search function of the present invention;

[0055] Figure 10 Schematic diagram of the radar point cloud three-dimensional coordinate system of the present invention;

[0056] Figure 11 is a schematic diagram of the positioning point, vehicle and target position of the present invention;

[0057] Figure 12 It is a coordinate transformation flow chart of the present invention;

[0058] Figure 13 This is a diagram showing the detection effect of the RTAB-Map algorithm of the present invention;

[0059] Figure 14 This is the obstacle avoidance and navigation effect diagram of the present invention;

[0060] Figure 15 It is the SLAM and navigation flow chart of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0062] The present invention comprises two systems, the radar base station end and the vehicle end. The system operates as follows: Figure 1 As shown;

[0063] The radar base station consists of sensors and computing platforms, such as Figure 2 The sensor shown is a combination of a lidar and a camera. The camera and lidar are rigidly connected, and this connection is not easily deformed. Therefore, the data obtained through the radar-camera joint calibration can be highly available.

[0064] like Figure 3This diagram shows the logic for connecting the radar sensor and the computing platform. The sensor connects to the computing platform via Ethernet or USB, allowing it to transmit collected data. The computing platform is a standard x86 computer that calculates and stores sensor data. The computing platform is also equipped with a wireless communication module to wirelessly transmit target data to the vehicle.

[0065] like Figure 4 As shown, a multi-terminal target recognition and positioning method of multi-sensor fusion includes the following steps:

[0066] Step 1: Stack preprocessing of camera data and lidar data.

[0067] The stacking process is described as follows:

[0068] The lidar is in a fixed position, so its displacement does not need to be considered. The stacking process on the base station side is a loop. Each loop obtains sensor data and puts it into a queue. At the same time, if the queue reaches the set number of stacks, the oldest data is removed from the queue. For the two sensor sources, camera and radar, storage queues are established separately to separate the calculation process and data storage process of the two.

[0069] like Figure 5 As shown, several items are designated as items in the short-time and long-time stacks. The system will first perform stack processing on the items in the short-time stack and generate the results of the short-time stack. Then, the system will retrieve the minimum long-time stack containing the items in the short-time stack, and use the results of the short-time stack and other data items to generate the required stack results. After completing all stack operations, the system will send out all stack results.

[0070] For example, in terms of the camera, a low-exposure data and a high-exposure data are needed. Assuming there are 50 data in total, 10 are stacked into the low-exposure data needed, and 50 are stacked into the high-exposure data. Then stack 10 first, and then copy the result and stack it with the remaining 40.

[0071] When new sensor data (blue data) arrives, the system puts it into the queue. If the queue reaches the set stack number, the oldest data (red data) will be discarded and the stacking process will continue. After sending the stack result, wait for the next frame of data to arrive.

[0072] By applying this stacking process to the point cloud scanned by the radar, a denser point cloud can be acquired at a higher refresh rate, thereby speeding up the response of the entire system.

[0073] By applying this stacking process to the camera data, we can obtain both high-exposure and low-exposure images at the same time, which to some extent solves the overexposure problem of the camera.

[0074] Step 2: Time synchronization and jitter prevention;

[0075] A unique time code is obtained based on the system UNIX clock as the basis for time synchronization. At the same time, when processing sensor data, due to the use of multi-threaded processing, the data may be jittered due to the different complexity of the data obtained by each thread, which will affect the control.

[0076] Therefore, when collecting raw data, a structure is used to encapsulate the data and timestamp and send it to the queue. At the same time, when processing the data, the timestamp is used for time synchronization. By detecting the difference between the timestamps of the point cloud data and the image data, if it is greater than a certain threshold, the data on the older side is discarded and new data on that side is waited for to be transmitted. If the requirements are still not met after waiting for the specified number of rounds, all the data of the thread is discarded and new data is obtained.

[0077] To prevent jitter, a checker is set before communication is sent to compare the timestamp of the data frame with the latest time saved in the checker. If the time is earlier than the latest time, it is considered that the round of data processing has timed out. At this time, the data frame is discarded and waits for new data to be obtained; if the time is later than the latest time, the data frame is encoded and sent, and the latest time in the checker is updated with the timestamp.

[0078] At the same time, when processing the merged data, a timeliness checker is set before and after some time-consuming steps. By obtaining the latest time saved in the sending checker, it is checked whether there is a timeout. If the current time is earlier than the latest time, it is considered that the remaining processing process is meaningless. At this time, the thread is directly destroyed to reduce resource overhead.

[0079] Step 3: Build a neural network for target detection;

[0080] like Figure 6 Regarding the neural network operation process, the first layer of the neural network uses the PicoDet network to identify the vehicle body and the positioning points on the site, and the location of the positioning points is identified through the first layer of the neural network; the second and third layers are for the refinement of the vehicle body. If the obtained label is the vehicle body, the ROI is extracted and the image is cropped before being sent to the second layer of the network; the second layer of the neural network uses the identification point model trained by the Yolov3 network to mainly identify the location of the identification points on the vehicle, and after identifying the corresponding ROI, it is sent to the third layer of the network; the third layer of the network performs one-stop recognition and classification of the color and digital labels of the ROI prepared in the second step, and then determines the vehicle information; the CNN network is used for classification in the third layer;

[0081] At the same time, there is an alternative to the third layer neural network, which can realize the color recognition of the identification point through brightness threshold and HSV threshold screening, and realize the recognition of the vehicle's digital label through SVM or digital recognition neural network. The recognition effect is as follows: Figure 7 shown.

[0082] Step 4: Image and radar information fusion and joint search: fuse the point cloud with the image and search for objects on the image;

[0083] like Figure 8 For the fusion and joint search flow chart, target detection is based on two-dimensional data collected by industrial cameras. To improve accuracy in applications, lidar is introduced to obtain accurate depth information of objects. LiDAR can obtain spatial point cloud information. After joint calibration of the camera and lidar, projection transformation can be used to find points in the lidar point cloud from points in an image. This method is multi-sensor fusion.

[0084] The principle of multi-sensor fusion is to transform the laser radar coordinate system into the camera coordinate system through rotation and translation through rigid coordinate transformation. This can establish a transformation from a plane pixel to a 3D point in the point cloud, and thus obtain the spatial position of the region of interest identified in the camera image.

[0085] The rotation matrix R is obtained by joint calibration 3×3 and the translation matrix t 3×1 , therefore, each point in the three-dimensional point cloud is converted into Transformed into a point in the camera coordinate system The transformation formula is:

[0086]

[0087] The transformation process is represented by a conversion matrix:

[0088]

[0089] Since the camera lens has distortion and offset, it is necessary to introduce the camera's internal parameter information for correction;

[0090] For a coordinate point [x c ,y c ,z c ], use the following matrix to transform it into a point [x m ,y m ,1]:

[0091]

[0092] Among them, c x ,c yrepresents the displacement from the imaging plane to the pixel plane, f x ,f y It is related to the focal length of the camera and together they constitute the intrinsic parameters of the camera.

[0093] There are two types of camera distortion: radial and tangential. Radial distortion is corrected by parameters k1, k2, and k3, while tangential distortion can be corrected by parameters p1 and p2. The formula for correcting the original coordinates (x, y) of the camera without any distortion to the actual imaging coordinates (x0, y0) of the camera is as follows, where (r, d) is the polar coordinate form of (x, y):

[0094]

[0095] like Figure 9 , project the 3D point cloud to obtain a mapping table of all spatial points to the plane. Note that the number of points in the point cloud is often large, and the traversal cost is too high, so the point cloud is cropped; for the neural network to search for a specific target, the set of points in the point cloud that may constitute the target is screened out;

[0096] Call numpy's logicand function to traverse all the obtained point information, determine the spatial position of each point one by one, and obtain a set of True or False lists of equal length; use this list as a mask to filter the original point cloud and projection results to obtain a simplified version of the point cloud and the corresponding lookup table, which can reduce the cost of subsequent point search operations.

[0097] Note that not every point in the image has a corresponding point on the point cloud. The image coordinates are appropriately expanded to form a larger search box, and then filtered and segmented to obtain a list of required points.

[0098] Using the clustering averaging method, the point set constituting the target object is converted into an approximate point in space. This process can be encapsulated into a search function, which accepts a pixel coordinate (x p ,y p ), expand it by the preset expansion amount to obtain an aggregated world coordinate (x w ,y w ,z w );

[0099] Step 5: Spatial position calculation;

[0100] After having a method to obtain the spatial position of the target of interest, the pixel coordinates of the target can be identified with the help of a neural network, and then the spatial position of the target can be obtained. If you want the robot to determine its moving target, you can use several spatial vectors to solve the relative position.

[0101] like Figure 10As shown, the radar defines a coordinate system O-XYZ for the point cloud coordinates it publishes, and the Z axis of the coordinate system points to the top surface of the lidar;

[0102] The origin of the point cloud coordinate system is regarded as the origin of the world coordinate system. The coordinates X1 = (x1, y1, z1) and X2 = (x2, y2, z2) under the radar point cloud coordinate system are obtained through the search function, thereby constructing a spatial vector pointing from the origin to the two coordinates. Then construct a space vector from X1 to X2

[0103]

[0104] In a given venue, set up Figure 11 The two positioning points shown in the figure require that the line connecting the two points is at a 45-degree angle to the edge of the field. ° The oblique intersection makes the airborne IMU and the host computer close to the longitudinal edge of the site when powered on, so that it can remember the direction of the site edge as the initial rotation angle (zero value), unifying the radar and vehicle coordinate systems in direction; according to the actual task requirements, it is considered that the vehicle, target and positioning point are approximately on the same plane, with the vehicle as the pole, the vector pointing to the far end of the site and parallel to the longitudinal edge as the polar axis, and the clockwise direction as the positive direction, a polar coordinate system ρ-r(θ) on the horizontal plane can be established; at the same time, the vehicle's airborne SLAM program uses Mapping begins with this point as the origin, obtaining the coordinate system O1-XYZ; SLAM is used for position estimation, and the IMU is integrated to obtain the attitude conversion information from the vehicle's own coordinate system O2-XYZ to the SLAM mapping coordinate system O1-XYZ. The polar coordinate information (d, θ) of the target point on the polar coordinate system ρ-r(θ) is then converted to the point [X, Y, 0] on the XY plane in the vehicle coordinate system O2-XYZ. This achieves the unification of the radar coordinate system and the vehicle coordinate system, as well as the target point searched by the radar and the vehicle target point.

[0105] After identifying the far and near positioning points through the neural network, a spatial vector is constructed from the closer positioning point to the farther positioning point. As a reference vector, theoretically, the polar vector can be obtained by rotating this vector 45° counterclockwise in the xy plane. However, in actual application, the xy plane of the radar coordinate system O-XYZ is not necessarily parallel to the horizontal plane, which makes it difficult to determine the rotation parameters of the vector. Therefore, the rotation process is achieved by introducing compensation after obtaining the vector angle.

[0106] After determining the spatial position of the vehicle and the target point, a three-dimensional vector pointing from the vehicle to the target point is constructed. The distance d between the vehicle and the target is obtained by solving the modulus of the vector; for the direction angle solution, the three-dimensional vector With reference vector The angle formula is obtained, and the positive and negative direction angle is obtained by the vector obtained by the cross product of the two vectors The positive or negative value of z determines the clockwise rotation angle θ of the two vectors from a bird's-eye view. The formula is as follows:

[0107]

[0108] Then the position (d, θ) of the target in the polar coordinate system ρ-r(θ) is obtained, thereby uniquely determining the target position in the vehicle coordinate system and realizing the transformation from the radar coordinate system to the vehicle coordinate system.

[0109] After confirming the target position, the target position information is sent to the vehicle host computer using wireless communication. After coordinate conversion, the navigation target point is uniquely determined in the SLAM map coordinate system O1-XYZ. The coordinate transformation process is as follows: Figure 12 shown.

[0110] Step 6: Visual SLAM is used for mapping and navigation to achieve robot obstacle avoidance and path planning. Visual SLAM is used on the robot side, which has lower sensor accuracy requirements than traditional SLAM. This is because high-precision maps rely on fixed lidar, so the vehicle-side SLAM only needs to perform simple obstacle avoidance and path planning based on the local map.

[0111] The open-source RTAB-Map algorithm is used for mapping, and the open-source Move_Base algorithm is used for path planning. The coordinate transformation of the computing platform data is described as follows:

[0112] On the vehicle side, since the vehicle and the target point are on the same plane, after obtaining the current posture information of the vehicle through the RTAB-Map mapping algorithm carried by the vehicle, the deflection angle between the current direction of the vehicle and the initial direction can be obtained. By adding this deflection angle to the received polar coordinates, the two-dimensional polar coordinates can be converted into a point on the XY plane of the spatial coordinate system with the current vehicle position as the origin; the conversion of vehicle coordinates to global SLAM map coordinates is managed by ROS, and the coordinates of the target point are determined on the map after the transformation with the help of the coordinate transformation function of ROS.

[0113] The vehicle-side sensor collects binocular camera images. The binocular camera has a binocular estimation function and uses the PnP function to estimate the depth and obtain relatively accurate depth and color information at close range.

[0114] The purpose of the RTAB-Map algorithm is to provide an appearance-based positioning and mapping solution that is independent of time and scale. This solution is aimed at solving the problem of online closed-loop detection in large environments. The idea of ​​the solution is that in order to meet real-time constraints, only a limited number of positioning points are used for closed-loop detection, while at the same time, the positioning points of the entire map can be accessed when needed; when the number of positioning points in the map makes the time to find a positioning match exceed a certain threshold, RTAB-Map transfers the positioning points in the WM that are unlikely to form a closed loop to the LTM, so that these transferred positioning points do not participate in the next closed-loop detection operation. When a closed loop is detected, its adjacent positioning points can be retrieved from the LTM and put into the WM for future closed-loop detection. The detection effect is as follows: Figure 13 shown.

[0115] The navigation function uses the Move_base algorithm in the navigation of the ROS system to create local cost maps and global cost maps for obstacle avoidance and navigation. The effect is as follows Figure 14 As shown;

[0116] SLAM and navigation process Figure 15 shown.

[0117] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A multi-sensor fusion multi-terminal target recognition and positioning method, characterized in that: The following steps are involved: Step 1: Preprocess the camera data and lidar data by stacking; The stack obtains sensor data through a receiving loop, first stacking the items of the short-term stack and generating the corresponding results; Secondly, the minimum long-term stack containing the items in the short-term stack is retrieved, and the required stack result is generated using the result of the short-term stack and the remaining item data; finally, the stack operation is ended and the stack result is sent out; Step 2: Time synchronization and jitter prevention; Step 3: Build a neural network for target detection; Step 4: Image and radar information fusion and joint search: fuse the point cloud with the image and search for objects on the image; Step 4 specifically includes: S41, obtaining a rotation matrix and a translation matrix through joint calibration, and transforming each point in the 3D point cloud into a point in the camera coordinate system, and representing the transformation process with a transformation matrix; S42, correcting the original coordinates of the camera to the actual imaging coordinates of the camera; S43. First, project the 3D point cloud to obtain a mapping table of all spatial points to the image plane, search and extract points within the actual image range, and delete those outside the camera image range; Secondly, the neural network searches for a specific target and selects the set of points in the point cloud that constitute the target; S44, using a cluster averaging method to convert the point set constituting the target object into an approximate point in space, and encapsulating the point set into a search function, and accepting a pixel coordinate range through the search function and expanding it using an expansion amount to obtain an aggregated world coordinate; Step 5: Perform spatial position calculation; Step 5 specifically includes: S51. Obtain the coordinates of two points under the radar point cloud coordinates through the search function , , construct a space vector pointing from the origin to the two coordinates; S52, unify the radar and vehicle coordinate systems in the direction and establish a polar coordinate system on the horizontal plane; S53, constructing the spatial vectors of the near and far positioning points; constructing the spatial vector from the vehicle to the target point, and calculating the distance between the vehicle and the target; S54, calculating the direction angle using the angle formula of the two space vectors in step S53; S55, then convert the polar coordinate information of the target point in the polar coordinate system into the vehicle coordinate system Points on a plane; S56, obtaining the position of the target in the polar coordinate system; S57. Use SLAM to estimate the position and fuse the IMU to obtain the posture conversion information from the polar coordinate point to the vehicle coordinate system and then to the SLAM mapping coordinate system.

2. The multi-terminal target recognition and positioning method of multi-sensor fusion according to claim 1 is characterized in that: Step 2 specifically includes: using a structure to encapsulate the data and timestamp and sending them to the queue. At the same time, when processing the data, the timestamp is used for time synchronization; by detecting the difference between the timestamps of the point cloud data and the image data, if it is greater than a certain threshold, the data on the older side is discarded and new data on that side is waited for to be transmitted; if the requirements are still not met after waiting for the specified number of rounds, all thread data is discarded and new data is obtained.

3. The multi-terminal target recognition and positioning method of multi-sensor fusion according to claim 1 is characterized in that: Step three specifically includes: the first layer uses the PicoDet network to identify the vehicle body and positioning points, extracts the ROI from the vehicle body and performs image cropping; the image is sent to the second layer of the Yolov3 network to identify the position of the identification points; the third layer uses the CNN network to identify and classify the color and digital labels of the ROI identification features.

4. The multi-sensor fusion multi-terminal target recognition and positioning method according to claim 3, characterized in that: The third layer also includes brightness threshold and HSV threshold screening to achieve color recognition of the marking points.

Images