An unmanned vehicle target guiding and tracking integrated visual detection platform and method

Abstract

The application discloses an unmanned vehicle target guiding and tracking integrated visual detection platform and method, which aims at target visual guiding and tracking, and comprises a target visual detection and prediction system based on YOLO-Kalman, a communication mode based on TCP / IP combined with ROS, and a GUI man-machine interactive interface; the target visual detection and prediction system based on YOLO-Kalman carries out prior training and feature detection on a specific target through algorithms such as YOLOv7, and predicts the coordinates of the target at the next moment through Kalman filtering; the communication mode based on TCP / IP combined with ROS imports the target information detected and predicted into the system through a TCP / IP protocol, and controls and reads the chassis information, GPS information and the like of the unmanned vehicle through a ROS robot operation platform in the system; the GUI man-machine interactive interface is integrated to reduce the use difficulty of users, and facilitates a series of tasks such as selection of a visual detection version, selection of a guiding target, tracking, acquisition of geographical coordinates and the like. The application meets the general demand of visual guiding in complex conditions.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to an integrated visual detection platform and method for target guidance and tracking. Background Technology

[0002] With the continuous advancement of target detection technology, target detection and guidance methods applied to autonomous vehicle platforms have also made significant progress. However, current visual guidance robot platforms face challenges such as limited application scenarios for target detection methods, limited platform communication methods, low system fluency, and cumbersome operation. Therefore, combining emerging target detection and search algorithms to better and faster complete target detection and guidance tasks, and constructing integrated and sophisticated system platforms, has become the mainstream of related research.

[0003] Research on vision-based unmanned mobile platforms is increasing, but most focuses on template matching or QR code visual search. While these methods require less computing power and offer higher accuracy, they impose stringent requirements on the shape of the target and cannot achieve omnidirectional visual search. They also fail to meet practical needs in complex backgrounds where targets are relatively uncertain. The YOLO series of algorithms utilizes a single-stage object detection algorithm based on convolutional neural networks (CNNs), using grid detection to achieve regression prediction of targets. Recent versions, YOLOv5, YOLOv6, and YOLOv7, can be trained using residual networks to quickly and accurately locate and predict targets in images. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides an integrated visual detection platform and method for target guidance and tracking of unmanned vehicles. This system, designed for target visual guidance and tracking, includes a YOLO-Kalman-based target visual detection and prediction system, a TCP / IP-based communication method combined with ROS, and a GUI (User-Defined Interface). The YOLO-Kalman-based target visual detection and prediction system uses algorithms such as YOLOv7 to perform pre-training and feature detection on specific targets, and predicts the target's coordinates at the next moment using Kalman filtering. The TCP / IP-based communication method imports the detected and predicted target information into the system via the TCP / IP protocol, and the ROS robot operating platform within the system controls and retrieves information such as the unmanned vehicle's chassis and GPS data. The integrated GUI reduces the difficulty of use for users, facilitating tasks such as selecting the visual detection version, selecting and tracking the guided target, and obtaining geographical coordinates. This invention meets the general needs of visual guidance in complex situations.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] An integrated visual detection platform for target guidance and tracking of unmanned vehicles includes a target visual detection and prediction system based on YOLO-Kalman, a communication module based on TCP / IP combined with ROS, and a GUI human-computer interaction interface module.

[0007] The target visual detection and prediction system based on YOLO-Kalman includes YOLO detection and Kalman prediction; it completes target detection of streaming video frames through the YOLO target detection algorithm, and after returning the coordinate center point, it enters a five-layer Kalman filter to complete the filtering of the target point and the prediction of the next state;

[0008] The YOLO detection integrates three independent YOLO versions, YOLOv5, YOLOv6, and YOLOv7, along with corresponding unmanned mobile platform guidance packages. Images of the target are recorded under different backgrounds and angles. After obtaining streaming video images, frames are read and cropped. LabelImg is used to perform labeling operations to obtain the target dataset. After obtaining the YOLO-formatted coordinate information, the YAML file is modified and a training path is created. Weights and network training parameters are selected, and the three YOLO versions are trained.

[0009] After training, the video information is transmitted via the USB port through the visual sensor. The acquired visual matrix is ​​input to three versions of the YOLO detection program respectively, the video is read and displayed in real time, and after target detection, the image information, prediction accuracy information and target center point information are returned. The target center point information is then smoothed and filtered.

[0010] After obtaining the continuous motion state of the target, the Kalman filter is introduced, and the target detection results of the last few frames are used as the observation values ​​of the Kalman filter to obtain the predicted value of the center point at the next moment.

[0011] The YOLO object detection algorithm converts the coordinates of the corner points of the input image bounding box into the coordinates of the center point and returns them to the main detection function. In the main detection function, a list is created to import and store the returned center coordinate values, and the list data is then passed to the Kalman filter. After obtaining the target center point coordinates within a set time period, the Kalman filter is divided into two modules: prediction and update. In the prediction module, the predictor estimates and makes a prediction based on the previous target detection. The update module calculates the measurement margin, measurement margin covariance, and optimal Kalman gain, which are used to update the filter, and this process is iterated multiple times to output the coordinate prediction for the next time step.

[0012] The communication module based on TCP / IP and ROS enables platform information transmission, including a TCP / IP-based visual communication protocol and a sensor software multiplexing platform based on the ROS operating system. In the TCP / IP-based visual communication protocol, a server / client is built to transmit the coordinate positions in the streaming video frames and the predicted results in the system. In the ROS communication mode, the unmanned platform drive mechanism and GPS information are published through ROS message information.

[0013] The TCP / IP-based visual communication protocol includes a TCP client-server port communication submodule, ROS topic and service communication methods;

[0014] The TCP client and server port communication submodule defines the client sending requests as the camera image environment in the YOLO environment of Python 3; the server responding is the mobile robot chassis drive command in the ROS environment of Python 2; the ROS environment's Socket function is initiated, and constraints are imposed through the binding function, that is, binding the current information symbol and IP; the Listen function is used to declare the server file representation, and finally the Accept connection function is blocked; after the connection is established, the TCP / IP protocol provides full-duplex communication for simultaneous data writing on the same link at the same time, and the system program is terminated through the TCP four-way handshake;

[0015] The ROS topic and service communication method receives and decodes cross-system TCP / IP messages under the same program file to obtain the target coordinate location. At this time, the ROSCORE terminal node is started and the motor drive topic information is published and GPS geographic coordinate data is subscribed to to complete the overall drive.

[0016] Transmit the real-time coordinates and predicted coordinates of the target center point fed back from the YOLO detection-Kalman prediction program.

[0017] GPS geographic coordinate data publishing follows the unified standard protocol NMEA, collecting current time, latitude and longitude information, number of satellites calculated, horizontal accuracy factor, and altitude.

[0018] The method for publishing and reading GPS geographic coordinate data is as follows: GPS information data is encapsulated into a message format and integrated into a publisher for message publishing. The receiving end subscribes to topic messages and exports latitude and longitude in the class.

[0019] The GUI human-computer interaction interface module uses EASYGUI in Python to display the platform client interface, including the selection of YOLO version, the selection of training weights, the selection of visual sensor interface, the selection of guidance target, and the start of communication system.

[0020] Preferably, the three YOLO versions are implemented using the conda virtual environment under Anaconda during environment configuration, and all three conda virtual environments are set to conda activate=3.8 when activated.

[0021] Preferably, the server is running in a virtual environment activated by Anaconda using Python 3, the client is running in a Python 2 script on the ROS platform operating system, and the robot's ROS system runs on Ubuntu Bionic 18.04.

[0022] Preferably, the unmanned mobile platform adopts four-wheel drive differential steering control and uses a serial port as the communication interface.

[0023] Preferably, the unmanned mobile platform is connected to the system computer via an RS232 serial port. The serial port publishes commands with a baud rate of 115200 and uses hexadecimal for transmission. The verification method uses XOR check to ensure the consistency of data transmitted between the host computer and the slave computer.

[0024] An integrated visual detection method for target guidance and tracking of unmanned vehicles specifically includes the following steps:

[0025] Step 1: Design the YOLO target set and detection program to achieve target detection;

[0026] Step 1-1: Create a target dataset for the target images, including collecting images of the target against different backgrounds, labeling the target, selecting pre-training weights and training the network, and generating the final optimal target weights;

[0027] Steps 1-2: Design a YOLO-based target detection module for streaming video. Select YOLOv5, YOLOv6, and YOLOv7 target detection. Activate the python=3.8 environment in Anaconda and install the relevant configuration files in sequence to complete the reading of streaming video frames.

[0028] Steps 1-3: Design the types of video detection in the integrated visual detection platform for target guidance and tracking, modify the relevant YAML files and weight files in the detection program to enable the detector to read target information and match, and set the detection step size and confidence level;

[0029] Step 2: Modify the YOLO detector in conjunction with the Kalman prediction program;

[0030] Step 2-1: Write a Kalman filter, define the state transition matrix, control input matrix and QR matrix, and define the initial state to conform to the input-output coordinate form;

[0031] Step 2-2: In the Kalman filter of Step 2-1, the mean square error array is first initialized, and then the state prediction, mean square error prediction, filter gain weight and filter mean square error update matrix are performed in sequence through a finite time loop, and finally the state prediction is output.

[0032] Steps 2-3: In the plot function of the utils of the detect program index in different YOLO versions, define and modify the target detection output function, create a target trajectory coordinate list based on the target label, find the target center point through the image corner points obtained by target detection, save the two parameters in the center point coordinates (x, y) to the trajectory coordinate list and refresh it, use the pop function to update and iterate, and only save the last 30 sets of coordinate data in the list;

[0033] Step 2-4: Substitute both sets of parameters from the center point coordinates (x, y) in the list into the Kalman filter in Step 2-1, iterate 5 times, and use the Kalman filter algorithm to obtain the predicted coordinates.

[0034] Steps 2-5: Design coordinate return functions to export the real-time coordinate values ​​and Kalman predicted coordinate values ​​respectively;

[0035] Step 3: Design TCP / IP communication based on WebSocket;

[0036] Step 3-1: First, create a TCP / IP client, import the socket module in Python, establish a connection to the server and set up a 4-bit serial port. After the port is bound, send the send test command and wait to write and read the coordinate information.

[0037] Step 3-2: Based on the YOLO-Kalman predictive visual detection program, build a TCP / IP server. Following the same steps as in Step 3-1, after importing the relevant modules, construct the same serial port number according to the client's requirements, bind the socket and port number, and start continuously listening for the send information mentioned in Step 3-1. The received request is read through accept, and after sending the target detection and prediction program information, wait to close the socket.

[0038] Step 3-3: Test the connection. Open the server and client in sequence and parse the channel transmission information. At this time, the transmission information is a binary data stream. The required coordinate information can only be transmitted through binary-to-string conversion.

[0039] Step 4: In conjunction with TCP / IP communication, embed the ROS operating system in the client for other communication purposes;

[0040] Step 4-1: On the Ubuntu 18.04 system, select the Melodic version of ROS, install the RQt software, and import Python 2.7 and related packages;

[0041] Step 4-2: In the client programming in Step 3-1, Python 2 is selected to create the unmanned vehicle chassis control node handle and to test the ROS information command release to the motor;

[0042] Step 4-3: Start the ROSBAG package to record coordinate location information and prediction information, and use it in conjunction with RQT to draw the trajectory graph;

[0043] Step 5: Import the GPS position coordinate reading module based on the NMEA protocol into the integrated visual detection platform based on target guidance and tracking;

[0044] Step 5-1: Parse the GGA global positioning system positioning data general statement in the NMEA navigation standard protocol, and parse the geographic coordinates and corresponding timestamps;

[0045] Step 5-2: Place the signal booster station, call the nmea_navsat_driver package in the ROS system, and set the USB connection port and baud rate;

[0046] Step 6: Information fusion between the two communication methods on the platform;

[0047] Step 6-1: Establish a connection via TCP / IP three-way handshake. First, after the image corner information is integrated by the system, the image processing computer sends a connection request data packet to the mobile robot system platform. Next, the mobile robot platform sends a connection agreement and synchronization request to the image computer. Finally, the image processing computer sends an acknowledgment to the mobile robot to accept the synchronization, thereby establishing a TCP remote connection, and the two hosts can officially send data.

[0048] Step 6-2: Release the connection via TCP / IP four-way handshake. After the message is sent, the image computer, in order to release the connection, will send a no-data segment (FIN set to 1) to the mobile robot and close the connection. The mobile robot will return an acknowledgment message upon receiving it. At this point, the image computer will be unable to send any more messages. The image computer will randomly enter a waiting state. At this time, the robot platform will send a connection release request, with FIN also set to 1. The image computer will receive the acknowledgment message, enter the TIME_WAIT waiting state, and close the TCP connection.

[0049] Step 6-3: From the perspective of visual sensing, after acquiring RGB image information through a visual sensor, a verifiable YOLO image is obtained through streaming video image conversion, and YOLO target detection is performed; after the Kalman filter completes the model target prediction, the coordinate information of the image center point, i.e. the detection point, is obtained, and the position and attitude information of the target in the world coordinate system is obtained through the coordinate transformation in Section 3.3.

[0050] Step 7: Complete the development of the interactive interface for the integrated visual detection platform for target guidance and tracking of unmanned vehicles;

[0051] Step 7-1: In the root directory of object detection, create a GUI program file based on EASYGUI and set the continuity problem options;

[0052] Step 7-2: In the program file of Step 7-1, import the os module and grant it administrator privileges. For the different options captured, import them into the terminal console and publish the relevant information. The settings for the consecutive options are shown in Steps 7-3 to 7-8.

[0053] Step 7-3: The GUI interface first enters the welcome screen. Click "OK" to enter the system, or cancel to exit the program.

[0054] Step 7-4: In the GUI interface, you will be prompted to decide whether to perform vehicle guidance. Clicking "Yes" will open the ROS terminal and start the relevant vehicle motor driver program. Clicking "No" will only start the target detection program.

[0055] Step 7-5: Enter the YOLO version selection in the GUI interface. Double-click to select any of YOLOv5, YOLOv6, and YOLOv7 for object detection.

[0056] Step 7-6: Enter the vision sensor selection interface in the GUI interface. Double-click to confirm the selection of an external vision sensor controlled by a specific USB port or the computer's built-in camera.

[0057] Step 7-7: Enter the target set selection interface in the GUI interface. Double-click to select the target from the COCO dataset, the training target (car target), or the face feature target.

[0058] Steps 7-8: The GUI interface will enter the selection confirmation screen. Verify the previously selected information and click "OK" to start the system detection.

[0059] The beneficial effects of this invention are as follows:

[0060] This invention fully realizes the terminal guidance of mobile robot platforms using emerging target detection algorithms (YOLOv5, YOLOv6, YOLOv7), meeting the general needs of visual guidance in complex situations. The convenient option modules and the application of multiple sensors allow operators to quickly get started, combining visual sensors and GPS information to complete various guidance and tracking needs for specific targets. Attached Figure Description

[0061] Figure 1 This is a diagram of the overall platform structure of the present invention.

[0062] Figure 2 This is a flowchart of the GUI operation interface of the present invention.

[0063] Figure 3 This is a block diagram illustrating the construction of the communication method based on TCP / IP combined with ROS according to the present invention.

[0064] Figure 4 This is a flowchart of the target visual detection and prediction system based on YOLO-Kalman, according to the present invention.

[0065] Figure 5 This is a partial GUI interface option jump diagram of an embodiment of the present invention. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0067] To overcome the limitations of existing technologies, such as restricted detection scenarios, limited communication methods between sensor systems, and cumbersome operation, and to improve system robustness and expand detection scenarios to adapt to different task environments, this invention proposes an integrated visual detection platform for unmanned vehicle target guidance and tracking. This platform includes different versions of visual target detection and prediction, and tracking guidance for a multi-sensor unmanned vehicle platform. The platform uses YOLO series target detection algorithms combined with Kalman filtering to predict target coordinates. A TCP / IP system is selected for communication between the visual sensors and the unmanned vehicle control platform. The control platform integrates the ROS robot operating system for functional expansion, supplemented by target position filtering and prediction, and GPS positioning of the detection vehicle, forming a highly integrated robot visual guidance system. The integrated GUI interface reduces user difficulty, facilitating tasks such as selecting visual detection versions, selecting guided targets, tracking, and obtaining geographic coordinates, thus fulfilling the general needs of visual guidance in complex situations.

[0068] The technical solution adopted by this invention to solve its technical problem is as follows:

[0069] An integrated visual detection platform for target guidance and tracking of unmanned vehicles includes a target visual detection and prediction system based on YOLO-Kalman, a communication module based on TCP / IP combined with ROS, and a GUI human-computer interaction interface module.

[0070] The YOLO-Kalman-based target visual detection and prediction system integrates multiple target detection algorithms and imports the target detection center coordinates into the Kalman program to predict the coordinates at future times.

[0071] Regarding the selection of YOLO versions, this invention platform uses three independent YOLO versions: YOLOv5, YOLOv6, and YOLOv7, along with their corresponding unmanned mobile platform guidance packages. Different versions are configured using conda virtual environments under Anaconda to avoid interference from different dependent versions. All three conda virtual environments are activated with `conda activate = 3.8`.

[0072] The target dataset can be either imported from the pre-trained official dataset or trained based on specific target images. First, images of the target under different backgrounds and angles are recorded to obtain streaming video images. After reading the frames and cropping them to a suitable size, labelImg is used to perform labeling operations. After obtaining the YOLO format coordinate information, the YAML file is modified and a training path is created. After selecting appropriate weights and network training parameters, specific target training is started.

[0073] For target reading in streaming video, a streaming video reading program is introduced to suppress visual sensor jitter and input the acquired visual matrix into the corresponding version of the YOLO detection program for video reading and real-time display. The acquired target information (including coordinate points) is smoothed and filtered.

[0074] The YOLO detection-Kalman prediction algorithm includes Kalman filtering for predicting YOLO target detection coordinates. The basic idea of ​​the Kalman motion predictor is as follows: after obtaining the continuous motion state of the target, it is imported into the Kalman predictor, and the target detection results of the last few frames are used as the observation values ​​of the Kalman filter to obtain the correction result of the next frame;

[0075] The Kalman filter algorithm for predicting YOLO target detection coordinates is defined as a class outside the detection function. The Kalman filter takes the target center point coordinates over a period of time as input. The class is mainly divided into two modules: prediction and update. In the prediction module, the predictor estimates and derives a prediction based on the previous target detection. In the update phase, the measurement margin, measurement margin covariance, and optimal Kalman gain are calculated and used to update the filter. This iterative process yields the predicted Kalman coordinates.

[0076] The YOLO detection-Kalman prediction algorithm modifies the YOLO object detection algorithm by modifying the single-object plotting function in the plot.py file under the utils folder linked by the detect script. This function converts the coordinates of the input image border corners into center point coordinates and returns them to the main detection function. In the main detection function, a list is created to store the returned center coordinate values. The list length is 30, and the pop(0) function is used to clear previous points during iteration. The list data is then passed to the Kalman filter function and iterated 5 times to obtain the predicted output at each coordinate time step.

[0077] This communication method, based on TCP / IP and ROS, includes TCP client and server port communication as well as ROS topic and service communication. It facilitates data transfer between different systems on the same port and between systems within the same system, offering strong stability and functional scalability.

[0078] The TCP client and server port communication method defines the client sending requests as the camera image environment in the YOLO environment of Python 3; the server responding is the mobile robot chassis drive command in the ROS environment of Python 2. The ROS environment's Socket function is used to create a socket, which is then bound by a binding function, i.e., binding the current information symbol to the IP address. At this point, the serial port's integrity must be ensured; if it is occupied, the binding will fail. Next, the Listen function is used to declare the server file representation. Finally, the Accept connection function is blocked. This will then trigger a three-way handshake, the specific process of which is as follows... Figure 3 As described above, after a connection is established, the TCP / IP protocol can provide full-duplex communication that allows simultaneous data writing on the same link at the same time. Finally, the system program is terminated through a TCP four-way handshake.

[0079] The ROS topic and service communication method receives and decodes cross-system TCP / IP messages under the same program file to obtain the target coordinate location. At this time, the ROSCORE terminal node is started and motor drive topic information is published, GPS geographic coordinate data is subscribed, etc., to complete the overall drive.

[0080] Based on the communication method of TCP / IP combined with ROS, the real-time coordinates and predicted coordinates of the target center point fed back in the YOLO detection-Kalman prediction program are transmitted.

[0081] The Kalman-YOLO detection and prediction program acts as the server, while the guidance node in the ROS robot operating system acts as the client. The client subscribes to the image coordinate information returned by the server and displays it to the guidance program. The guidance node in ROS also acts as a client, subscribing to GPS information and acting as a publisher, sending speed and displacement commands to the unmanned vehicle to drive the motor, thus completing the overall communication. The TCP / IP communication method is used, with the server in a virtual environment activated by Anaconda (Python 3 version), and the client in a Python 2 script in the ROS Melodic sub-platform operating system. The ROS system runs on Ubuntu Bionic 18.04.

[0082] TCP is a set of full-duplex industry standard protocols. To match the data transmission between the ROS Melodic operating system (written in Python 2) and the OpenCV 4 vision sensor (written in Python 3), and to simultaneously achieve communication between a single control board or multiple machines, this system developed a TCP / IP-based WebSocket module information transmission protocol. This allows computers with different CPU versions or operating systems to achieve real-time communication as long as they follow the same protocol (defining the server and client), facilitating loosely coupled control and subsequent multi-machine development.

[0083] GPS coordinates are published in accordance with the unified standard protocol NMEA (National Marine Electronics Association), which can collect current time, latitude and longitude information, calculate the number of satellites, horizontal accuracy factor and altitude;

[0084] For the ROS operating system in this platform, the GPS publishing-reading method is as follows: GPS information data is encapsulated into a message format and integrated into a publisher for message publishing. The receiving end subscribes to topic messages and exports latitude and longitude in the class.

[0085] For target guidance tasks of unmanned vehicle platforms, the mobile platform adopts four-wheel drive differential steering control and uses a serial port as the communication interface.

[0086] The mobile robot platform needs to connect to the system computer via an RS232 serial port. The serial port publishes commands with a baud rate of 115200 and uses hexadecimal for transmission. The verification method uses XOR check to ensure the consistency of data transmitted between the host computer and the slave computer.

[0087] The development of the GUI (Human-Computer Interface) aims to lower the barrier to entry, improve the efficiency of the system platform, and facilitate the expansion of new functions based on different options. The GUI continuously prompts users to select relevant information for target detection and guidance, reducing the difficulty of use and completing the overall platform construction.

[0088] An integrated visual detection platform for target guidance and tracking of unmanned vehicles, the specific usage of which includes the following steps:

[0089] Step 1: Design the YOLO target set and detection program to achieve target detection;

[0090] Step 1-1: Create a target dataset for the target images. The YOLO target dataset processing includes collecting images of the target under different backgrounds, creating labels for the target, selecting pre-training weights and training the network, and generating the final optimal target weights.

[0091] Steps 1-2: Design a streaming video target detection module based on the new version of YOLO, select the newly released YOLOv5 target detection, YOLOv6 target detection and YOLOv7 target detection, activate the python==3.8 environment in Anaconda, install the relevant configuration files in sequence, and complete the reading of streaming video frames;

[0092] Steps 1-3: Design the types of video detection in the integrated visual detection platform for target guidance and tracking, modify the relevant YAML files and weight files in the detection program to enable the detector to read target information and match it accurately, and set the detection step size and confidence level;

[0093] Step 2: Modify the YOLO detector in conjunction with the Kalman prediction program;

[0094] Step 2-1: Write a Kalman filter state estimator, define the state transition matrix, control input matrix and QR matrix, define the initial state, and make it input and output coordinate form;

[0095] Step 2-2: In the Kalman filter of Step 2-1, the mean square error array is first initialized, and then the state prediction, mean square error prediction, filter gain weight and filter mean square error update matrix are performed in sequence through a finite time loop, and finally the state prediction is output.

[0096] Steps 2-3: In the plot function of the utils of the detect program index in different YOLO versions, define and modify the target detection output function, create a target trajectory coordinate list based on the target label, find the target center point through the image corner points obtained by target detection, save the two parameters in the center point coordinates (x, y) to the trajectory coordinate list and refresh it, use the pop function to update and iterate, and only save the last 30 sets of coordinate data in the list;

[0097] Step 2-4: Substitute both sets of parameters from the center point coordinates (x, y) in the list into the Kalman filter state estimator in Step 2-1, iterate 5 times, and use the Kalman filter algorithm to obtain the predicted coordinates.

[0098] Steps 2-5: Design coordinate return functions to export the real-time coordinate values ​​and Kalman predicted coordinate values ​​respectively;

[0099] Step 3: Design TCP / IP communication based on WebSocket;

[0100] Step 3-1: First, create a TCP / IP client, import the socket module in Python, establish a connection to the server and set up a 4-bit serial port. After the port is bound, send the send test command and wait to write and read the coordinate information.

[0101] Step 3-2: Based on the YOLO-Kalman predictive visual detection program, build a TCP / IP server. Following the same steps as in Step 3-1, after importing the relevant modules, construct the same serial port number according to the client's requirements, bind the socket and port number, and start continuously listening for the send information mentioned in Step 3-1. The received request is read through accept, and after sending the target detection and prediction program information, wait to close the socket.

[0102] Step 3-3: Test the connection. Open the server and client in sequence and parse the channel transmission information. Note that the transmission information is a binary data stream at this time. We need to convert the binary data to a string to transmit the coordinate information we need.

[0103] Step 4: In conjunction with TCP / IP communication, embed the ROS operating system in the client for other communication purposes;

[0104] Step 4-1: In the Ubuntu 18.04 system, select the Melodic version of ROS, install necessary display software such as RQt, and import Python 2.7 and related packages;

[0105] Step 4-2: In the client programming in Step 3-1, Python 2 is selected to create the unmanned vehicle chassis control node handle and to test the ROS information command release to the motor;

[0106] Step 4-3: Start the ROSBAG package to record coordinate location information and prediction information, and use it in conjunction with RQT to draw the trajectory map.

[0107] Step 5: Import the GPS position coordinate reading module based on the NMEA protocol into the integrated visual detection platform based on target guidance and tracking;

[0108] Step 5-1: Parse the GGA global positioning system positioning data general statement in the NMEA navigation standard protocol, mainly parsing the geographic coordinates and the corresponding timestamp;

[0109] Step 5-2: Place the signal booster station, call the nmea_navsat_driver package in the ROS system, and set the USB connection port and baud rate;

[0110] Step 6: Information fusion for this platform in the two communication methods;

[0111] Step 6-1: Establish a connection via TCP / IP three-way handshake. First, after the image corner information is integrated by the system, the image processing computer sends a connection request data packet to the mobile robot system platform. Next, the mobile robot platform sends a connection agreement and synchronization request to the image computer. Finally, the image processing computer sends an acknowledgment to the mobile robot to accept the synchronization, thereby establishing a TCP remote connection, and the two hosts can officially send data.

[0112] Step 6-2: Release the connection via TCP / IP four-way handshake. After the message is sent, the image computer will send a message with no data (FIN set to 1) to the mobile robot to release the connection and close the connection. The mobile robot will return an acknowledgment message upon receiving it (at this time, the image computer cannot continue to send messages after receiving it). The image computer will randomly enter a waiting state. At this time, the robot platform will send a connection release request (FIN also set to 1). The image computer will acknowledge the request upon receiving it, enter a waiting state (TIME_WAIT), and close the TCP connection.

[0113] Step 6-3: From a visual sensing perspective, after acquiring RGB image information through a visual sensor, a verifiable YOLO image is obtained through streaming video image conversion, and YOLO target detection is performed. After the Kalman filter completes the model target prediction, the coordinate information of the image center point (detection point) is obtained. However, this information is pixel-level coordinate information. The position and attitude information of the target in the world coordinate system can be obtained through coordinate transformation in Section 3.3.

[0114] Step 7: Complete the development of the interactive interface for the integrated visual detection platform for target guidance and tracking of unmanned vehicles. Figure 2 );

[0115] Step 7-1: In the root directory of object detection, create a GUI program file based on EASYGUI and set the continuity problem options;

[0116] Step 7-2: In the program file of Step-1, import the os module and grant administrator privileges. For the different options captured, import them into the terminal console and publish the relevant information. The settings for the consecutive options are shown in Steps 7-3 to 7-8.

[0117] Step 7-3: The GUI interface first enters the welcome screen. Click "OK" to enter the system, or cancel to exit the program.

[0118] Step 7-4: In the GUI interface, you will be prompted to decide whether to perform vehicle guidance. Clicking "Yes" will open the ROS terminal and start the relevant vehicle motor driver program. Clicking "No" will only start the target detection program.

[0119] Step 7-5: Enter the YOLO version selection in the GUI interface. Double-click to select any of YOLOv5, YOLOv6, and YOLOv7 for object detection.

[0120] Step 7-6: Enter the vision sensor selection interface in the GUI interface. Double-click to confirm the selection of an external vision sensor controlled by a specific USB port or the computer's built-in camera.

[0121] Step 7-7: Enter the target set selection interface in the GUI interface. Double-click to select the target from the COCO dataset, the training target (car target), or the face feature target.

[0122] Steps 7-8: The GUI interface will enter the selection confirmation screen. Verify the previously selected information and click "OK" to start the system detection.

Claims

1. An integrated visual detection platform for target guiding and tracking of unmanned vehicles, characterized in that, It includes a target visual detection and prediction system based on YOLO-Kalman, a communication module based on TCP / IP combined with ROS, and a GUI human-computer interaction interface module; The target visual detection and prediction system based on YOLO-Kalman includes YOLO detection and Kalman prediction; it completes target detection of streaming video frames through the YOLO target detection algorithm, and after returning the coordinate center point, it enters a five-layer Kalman filter to complete the filtering of the target point and the prediction of the next state; The YOLO detection integrates three independent YOLO versions, YOLOv5, YOLOv6, and YOLOv7, along with corresponding unmanned mobile platform guidance packages. Images of the target are recorded under different backgrounds and angles. After obtaining streaming video images, frames are read and cropped. LabelImg is used to perform labeling operations to obtain the target dataset. After obtaining the YOLO-formatted coordinate information, the YAML file is modified and a training path is created. Weights and network training parameters are selected, and the three YOLO versions are trained. After training, the video information is transmitted via the USB port through the visual sensor. The acquired visual matrix is ​​input to three versions of the YOLO detection program respectively, the video is read and displayed in real time, and after target detection, the image information, prediction accuracy information and target center point information are returned. The target center point information is then smoothed and filtered. After obtaining the continuous motion state of the target, the Kalman filter is introduced, and the target detection results of the last few frames are used as the observation values ​​of the Kalman filter to obtain the predicted value of the center point at the next moment. The YOLO object detection algorithm converts the coordinates of the corner points of the input image bounding box into the coordinates of the center point and returns them to the main detection function. In the main detection function, a list is created to import and store the returned center coordinate values, and the list data is then passed to the Kalman filter. After obtaining the target center point coordinates within a set time period, the Kalman filter is divided into two modules: prediction and update. In the prediction module, the predictor estimates and makes a prediction based on the previous target detection. The update module calculates the measurement margin, measurement margin covariance, and optimal Kalman gain, which are used to update the filter, and this process is iterated multiple times to output the coordinate prediction for the next time step. The communication module based on TCP / IP and ROS enables platform information transmission, including a TCP / IP-based visual communication protocol and a sensor software multiplexing platform based on the ROS operating system. In the TCP / IP-based visual communication protocol, a server / client is built to transmit the coordinate positions in the streaming video frames and the predicted results in the system. In the ROS communication mode, the unmanned platform drive mechanism and GPS information are published through ROS message information. The TCP / IP-based visual communication protocol includes a TCP client-server port communication submodule, ROS topic and service communication methods; The TCP client and server port communication submodule defines the client sending requests as the camera image environment in the YOLO environment of Python 3; the server responding is the mobile robot chassis drive command in the ROS environment of Python 2; the ROS environment's Socket function is initiated, and constraints are imposed through the binding function, that is, binding the current information symbol and IP; the Listen function is used to declare the server file representation, and finally the Accept connection function is blocked; after the connection is established, the TCP / IP protocol provides full-duplex communication for simultaneous data writing on the same link at the same time, and the system program is terminated through the TCP four-way handshake; The ROS topic and service communication method receives and decodes cross-system TCP / IP messages under the same program file to obtain the target coordinate location. At this time, the ROSCORE terminal node is started and the motor drive topic information is published and GPS geographic coordinate data is subscribed to to complete the overall drive. Transmit the real-time coordinates and predicted coordinates of the target center point fed back from the YOLO detection-Kalman prediction program. GPS geographic coordinate data publishing follows the unified standard protocol NMEA, collecting current time, latitude and longitude information, number of satellites calculated, horizontal accuracy factor, and altitude. The method for publishing and reading GPS geographic coordinate data is as follows: GPS information data is encapsulated into a message format and integrated into a publisher for message publishing. The receiving end subscribes to topic messages and exports latitude and longitude in the class. The GUI human-computer interaction interface module uses EASYGUI in Python to display the platform client interface, including the selection of YOLO version, the selection of training weights, the selection of visual sensor interface, the selection of guidance target, and the start of communication system. 2.The unmanned vehicle target guiding and tracking integrated visual detection platform according to claim 1, characterized in that, The three YOLO versions are implemented using the conda virtual environment under Anaconda during environment configuration. When activating the three conda virtual environments, all three conda virtual environments are set to conda activate=3.

8. 3.The unmanned vehicle target guiding and tracking integrated visual detection platform according to claim 1, wherein, The server runs on a virtual environment activated by Anaconda using Python 3, while the client runs on a Python 2 script within the ROS Melodic subplatform operating system. The robot's ROS system runs on Ubuntu Bionic 18.

04.

4. The integrated visual detection platform for target guidance and tracking of unmanned vehicles according to claim 1, wherein, The unmanned mobile platform adopts four-wheel drive differential steering control and uses a serial port as the communication interface.

5. The integrated visual detection platform for target guidance and tracking of unmanned vehicles according to claim 1, wherein, The unmanned mobile platform is connected to the system computer via an RS232 serial port. The serial port publishes commands with a baud rate of 115200 and uses hexadecimal for transmission. The verification method uses XOR check to ensure the consistency of data transmitted between the host computer and the slave computer.

6. A method of detection using the visual detection platform of claim 1, wherein, Includes the following steps: Step 1: Design the YOLO target set and detection program to achieve target detection; Step 1-1: Create a target dataset for the target images, including collecting images of the target against different backgrounds, labeling the target, selecting pre-training weights and training the network, and generating the final optimal target weights; Steps 1-2: Design a YOLO-based target detection module for streaming video. Select YOLOv5, YOLOv6, and YOLOv7 target detection. Activate the python=3.8 environment in Anaconda and install the relevant configuration files in sequence to complete the reading of streaming video frames. Steps 1-3: Design the types of video detection in the integrated visual detection platform for target guidance and tracking, modify the relevant YAML files and weight files in the detection program to enable the detector to read target information and match, and set the detection step size and confidence level; Step 2: Modify the YOLO detector in conjunction with the Kalman prediction program; Step 2-1: Write a Kalman filter, define the state transition matrix, control input matrix and QR matrix, and define the initial state to conform to the input-output coordinate form; Step 2-2: In the Kalman filter of Step 2-1, the mean square error array is first initialized, and then the state prediction, mean square error prediction, filter gain weight and filter mean square error update matrix are performed sequentially through a finite time loop, and finally the state prediction is output. Steps 2-3: In the plot function of the utils of the detect program index in different YOLO versions, define and modify the target detection output function, create a target trajectory coordinate list based on the target label, find the target center point through the image corner points obtained by target detection, save the two parameters in the center point coordinates (x, y) to the trajectory coordinate list and refresh it, use the pop function to update and iterate, and only save the last 30 sets of coordinate data in the list; Step 2-4: Substitute both sets of parameters from the center point coordinates (x, y) in the list into the Kalman filter in Step 2-1, iterate 5 times, and use the Kalman filter algorithm to obtain the predicted coordinates. Steps 2-5: Design coordinate return functions to export the real-time coordinate values ​​and Kalman predicted coordinate values ​​respectively; Step 3: Design TCP / IP communication based on WebSocket; Step 3-1: First, create a TCP / IP client, import the socket module in Python, establish a connection to the server and set a 4-bit serial port. After the port is bound, send the send test command and wait for the coordinate information to be written and read. Step 3-2: Based on the YOLO-Kalman predictive visual detection program, build a TCP / IP server. Following the same steps as in Step 3-1, after importing the relevant modules, construct the same serial port number according to the client's requirements, bind the socket and port number, and start continuously listening for the send information mentioned in Step 3-1. The received request is read through accept, and after sending the target detection and prediction program information, wait to close the socket. Step 3-3: Test the connection. Open the server and client in sequence and parse the channel transmission information. At this time, the transmission information is a binary data stream. The required coordinate information can only be transmitted through binary-to-string conversion. Step 4: In conjunction with TCP / IP communication, embed the ROS operating system in the client for other communication purposes; Step 4-1: On the Ubuntu 18.04 system, select the Melodic version of ROS, install the RQt software, and import Python 2.7 and related packages; Step 4-2: In the client programming in Step 3-1, Python 2 is selected to create the unmanned vehicle chassis control node handle and to test the ROS information command release to the motor; Step 4-3: Start the ROSBAG package to record coordinate location information and prediction information, and use it in conjunction with RQT to draw the trajectory graph; Step 5: Import the GPS position coordinate reading module based on the NMEA protocol into the integrated visual detection platform based on target guidance and tracking; Step 5-1: Parse the GGA global positioning system positioning data general statement in the NMEA navigation standard protocol, and parse the geographic coordinates and corresponding timestamps; Step 5-2: Place the signal booster station, call the nmea_navsat_driver package in the ROS system, and set the USB connection port and baud rate; Step 6: Information fusion between the two communication methods on the platform; Step 6-1: Establish a connection via TCP / IP three-way handshake. First, after the image corner information is integrated by the system, the image processing computer sends a connection request data packet to the mobile robot system platform. Next, the mobile robot platform sends a connection agreement and synchronization request to the image computer. Finally, the image processing computer sends an acknowledgment to the mobile robot to accept the synchronization, thereby establishing a TCP remote connection, and the two hosts can officially send data. Step 6-2: Release the connection via TCP / IP four-way handshake. After the message is sent, the image computer, in order to release the connection, will send a no-data segment (FIN set to 1) to the mobile robot and close the connection. The mobile robot will return an acknowledgment message upon receiving it. At this point, the image computer will be unable to send any more messages. The image computer will randomly enter a waiting state. At this time, the robot platform will send a connection release request, with FIN also set to 1. The image computer will receive the acknowledgment message, enter the TIME_WAIT waiting state, and close the TCP connection. Step 6-3: From the perspective of visual sensing, after acquiring RGB image information through a visual sensor, a verifiable YOLO image is obtained through streaming video image conversion, and YOLO target detection is performed; after the Kalman filter completes the model target prediction, the coordinate information of the image center point, i.e. the detection point, is obtained, and the position and attitude information of the target in the world coordinate system is obtained through the coordinate transformation in Section 3.

3. Step 7: Complete the development of the interactive interface for the integrated visual detection platform for target guidance and tracking of unmanned vehicles; Step 7-1: In the root directory of object detection, create a GUI program file based on EASYGUI and set the continuity problem options; Step 7-2: In the program file of Step 7-1, import the os module and grant it administrator privileges. For the different options captured, import them into the terminal console and publish the relevant information. The settings for the consecutive options are shown in Steps 7-3 to 7-8. Step 7-3: The GUI interface first enters the welcome screen. Click "OK" to enter the system, or cancel to exit the program. Step 7-4: In the GUI interface, you will be prompted to decide whether to perform vehicle guidance. Clicking "Yes" will open the ROS terminal and start the relevant vehicle motor driver program. Clicking "No" will only start the target detection program. Step 7-5: Enter the YOLO version selection in the GUI interface. Double-click to select any of YOLOv5, YOLOv6, and YOLOv7 for object detection. Step 7-6: Enter the vision sensor selection interface in the GUI interface. Double-click to confirm the selection of an external vision sensor controlled by a specific USB port or the computer's built-in camera. Step 7-7: Enter the target set selection interface in the GUI interface. Double-click to select the target from the COCO dataset, the training target (car target), or the face feature target. Steps 7-8: The GUI interface will enter the selection confirmation screen. Verify the previously selected information and click "OK" to start the system test.

Images