Visual following method and control system of maintenance trolley

Abstract

The invention discloses a visual following method of a maintenance trolley and a control system, and the method comprises the steps: employing a generalized training YOLOv7 model and a visual positioning module based on a depth camera, combining a CSRT target tracking algorithm to position a tracking target in real time, and obtaining the spatial geometric position of the following target; and the spatial geometric position of the following target is used as input information of a motor control module, and the rotating speed and the torque of each motor of the trolley are controlled based on a double-closed-loop direct-current brushless motor speed regulation system. Compared with the prior art, the problems that the maneuvering speed change of the trolley is insensitive and overcurrent occurs at the starting moment can be solved; and in the target identification following process, the influence of the human trunk action on the depth information is reduced, and the target tracking precision is improved.

Description

(I) Technical field

[0001] The invention relates to a visual following method and a control system for a maintenance vehicle, and belongs to the technical field of visual recognition and the technical field of intelligent vehicles. (II) Background technology

[0002] At the equipment inspection and maintenance site, workers need to carry tools and accessories to designated locations. Traditional carts need to be towed by manpower, and at the inspection site where debris is piled up, there are many obstacles, which will further increase the difficulty of workers' carrying and reduce work efficiency. Therefore, target tracking can be used to assist in the transportation of tools and accessories, thereby improving the work efficiency of workers.

[0003] At present, the maintenance vehicles developed have problems such as insensitive motorized speed change and overcurrent at the moment of starting. During the image recognition process, the movement of the human body will have a certain impact on the depth information, and the target tracking accuracy is not high enough. (III) Summary of the invention

[0004] The present invention provides a visual following obstacle avoidance method and control system for a maintenance trolley, aiming to solve the problems of insensitive speed change of the trolley and instantaneous overcurrent during startup, reduce the influence of human trunk movement on depth information during image recognition, and improve target tracking accuracy.

[0005] The present invention provides a visual following obstacle avoidance method for a maintenance vehicle, comprising:

[0006] S1. The YOLOv7 model is further trained using image samples collected at the maintenance site to reduce the impact of human torso motion on depth information;

[0007] S2. Use the generalized trained YOLOv7 model and the visual positioning module based on the depth camera to locate and track the target in real time in a specific maintenance site environment in combination with the CSRT target tracking algorithm;

[0008] In a possible implementation, the YOLOv7 model is further trained using the image samples collected at the maintenance site, specifically including:

[0009] Acquire a target image training sample set through a depth camera, divide the target image training sample set into a training set and a verification set, further train the convolutional neural network based on the training set to obtain an initial target recognition model, and verify and optimize the initial target recognition model based on the verification set;

[0010] In a possible implementation, the CSRT target tracking algorithm specifically includes:

[0011] In a specific maintenance site environment, and when the target is within the specified tracking range, Kalman filtering technology is applied to predict the position and state of the target based on the dynamic model and observation data of the target. During the tracking process, new observation data will be continuously integrated into the target recognition model so that the model can be updated in real time to ensure that it can accurately reflect the evolution of the target's visual features and movement patterns;

[0012] In a possible implementation, the dual closed-loop brushless DC motor speed control system specifically includes:

[0013] The four motors are controlled by the dual closed-loop DC brushless motor speed control system, the inner loop is a current loop, which feeds back actual current information through a current acquisition circuit, and the outer loop is a speed measurement loop, which feeds back actual speed information through a photoelectric encoder;

[0014] The beneficial effects of the present invention are:

[0015] The present invention can further train the YOLOv7 model using image samples collected at the maintenance site, thereby reducing the influence of human body movements on depth information. The CSRT target tracking algorithm is adopted to provide higher target tracking accuracy. The DC brushless motor adopts the dual closed-loop DC brushless motor speed control system to improve the sensitivity of motorized speed change, increase the current cutoff loop, and solve the problem of instantaneous overcurrent when the motor starts. (IV) Description of the drawings

[0016] Figure 1 This is the system architecture diagram of the maintenance trolley;

[0017] Figure 2 It is the flow chart of the start-up and follow-up procedures of the maintenance trolley;

[0018] Figure 3 This is a distance calculation diagram of the visual following device of the maintenance trolley;

[0019] Figure 4 It is the principle block diagram of the dual closed-loop DC brushless motor speed control system. (V) Specific implementation methods

[0020] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] like Figure 2 As shown, Figure 2The present invention provides a flowchart of the maintenance vehicle startup follow-up procedure, and the method includes the following steps:

[0022] The generalized trained YOLOv7 model and the visual positioning module based on the depth camera are used to obtain the pixel coordinates of the follow-up target in the image, and the CSRT target tracking algorithm is combined to locate and track the target in real time. In this embodiment, before obtaining the real-time image, the system in the maintenance vehicle needs to be initialized, wherein the initialization settings include clock configuration, timer, interrupt initialization and serial port required for debugging the program. After initialization, the camera is calibrated to obtain the parameters required for modeling, namely the camera's intrinsic parameter matrix and extrinsic parameter matrix, wherein the intrinsic parameter matrix is ​​used for stereo correction of the image, and the extrinsic parameter matrix is ​​used for three-dimensional reconstruction of the image.

[0023] In this implementation scheme, a camera module is provided on the maintenance trolley, and real-time images are acquired through the camera module. Specifically, the camera is started and read frame by frame through the pipeline library in pyrealsense2, and real-time images and depth images are directly acquired by reading the camera module on the maintenance trolley.

[0024] In this embodiment, different angles are set, and the follow-up target is photographed from different angles to obtain portrait samples at different angles. At the same time, different picture sizes are set to obtain portrait samples of different picture sizes. By integrating portrait samples of different picture sizes and different shooting angles, a target image training sample set is obtained. Compared with photographing from a single angle and obtaining sample images of the same size, sample images of different angles and different picture sizes can further improve the accuracy of image recognition and avoid misrecognition caused by angle or picture scaling.

[0025] In this embodiment, the target image training sample set is divided into a training set and a verification set according to a preset ratio, the convolutional neural network is further trained based on the training set to obtain an initial target recognition model, and the initial target recognition model is verified and optimized based on the verification set to obtain a target recognition model.

[0026] In this implementation, the target is detected and visually located, the distance and angle of following the target are solved through the depth image, and the target is followed according to the solved information. The industrial computer processes the observation information and sends the speed and angle information changed according to the observation information to the motor control module through the serial port to control the movement of the car.

[0027] Furthermore, if Figure 3As shown, the CSRT target tracking algorithm is used to track the target in real time, and the depth image is used to obtain the distance R1 between the target and the small vehicle and the coordinates (x1, y1) of the target. The center of the image corresponds to the camera position, so the pixel coordinates of the center of the image are recorded as (x2, y2). According to the distance formula between two points in a two-dimensional plane:

[0028] R4=[(x1-x2) 2 +(y1-y2) 2 ] 1 / 2

[0029] Solve the distance R2 between the follow target and the camera center on the two-dimensional image, according to the inverse trigonometric function:

[0030] α=arcsin(R4 / R5)

[0031] The included angle α between the target and the car can be obtained, and the angle α and the distance R5 between the target and the car can be sent to the motor control module through serial communication to achieve real-time tracking of the target.

[0032] In this embodiment, the CSRT target tracking algorithm tracks the target within a limited tracking distance, and uses a Kalman filter to estimate the position and state of the target according to the motion model and observation information of the target. While tracking the target, the new observation information is integrated into the target recognition model, and the target recognition model is continuously updated to adapt to the changes in the appearance and motion state of the target;

[0033] Specifically, the CSRT target tracking algorithm includes:

[0034] CSRT uses the back-projection method when calculating likelihood. It uses the color information in the back-projection to capture the part close to the target color within the sampling range as the likelihood. According to the Bayesian posterior probability formula:

[0035] Given that h is the likelihood of the foreground, the prior probability P O and P b Under the premise of , we can find the posterior probability P P , so the probability map of the final required mask can be obtained, and the solution formula is:

[0036] p(m=1|y,x)∝(y|m=1,x).p(m=1) Where m is the learning reliability corresponding to each pixel. After solving the probability map of the mask, the new mask can be obtained, that is, the new position of the tracked target.

[0037] In this embodiment, if Figure 4As shown, the four motors are controlled by the dual closed-loop DC brushless motor speed control system. The inner loop is the current loop, which feeds back the actual current information through the current acquisition circuit. The outer loop is the speed measurement loop, which feeds back the actual speed information through the photoelectric encoder. The current position information of the maintenance trolley is used as the input of the trolley motor control module, and 4 pairs of complementary PWM waves are output. After passing through the H-bridge circuit, the speed and torque of each motor of the trolley are controlled.

[0038] Specifically, the dual closed-loop DC brushless motor speed control system includes: a speed setting link, a current setting link, a speed regulator ASR, a current regulator ACR, a DC brushless motor, such as Figure 4 As shown. The current feedback link uses a current acquisition circuit to measure the motor loop current and feeds it back to the ARM controller in the form of voltage. The armature current obtained after AD conversion is compared with the current given by the speed regulator ASR to obtain the current deviation, which is input to the current regulator ACR to control the PWM duty cycle, thereby forming a current inner loop. The speed is measured by a photoelectric encoder, and the result is fed back to the main control STM32F103IGT6, compared with the given speed, and the speed difference is obtained, and then the given current is controlled through the speed regulator ASR.

[0039] The use of current inner loop and speed outer loop dual closed loop control can achieve the purpose of current cutoff loop and motor quick start. When the current inner loop negative feedback is used, the output of the speed outer loop serves as the given of the current inner loop, and the purpose of current limiting is achieved by limiting the output of the outer loop.

[0040] During the motor starting process, the use of dual closed-loop control can enable the entire starting process to obtain the maximum current value and maximum torque starting. At this time, the speed increases in a linear manner, thereby reducing the time consumed by the motor starting and improving the sensitivity of motor speed change; the current cutoff loop effectively prevents the motor from generating a large impact current at the moment of starting, thereby effectively avoiding damage to power electronic devices.

[0041] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A visual following method for a maintenance vehicle, characterized in that: The method comprises the following steps: S1. The YOLOv7 model is further trained using image samples collected at the maintenance site to reduce the impact of human torso motion on depth information; S2. The generalized trained YOLOv7 model and the visual positioning module based on the depth camera are used to locate and track the target in real time in a specific maintenance site environment in combination with the CSRT target tracking algorithm.

2. A visual following method for a maintenance vehicle according to claim 1, characterized in that: The YOLOv7 model was further trained using image samples collected at the maintenance site, including: A training sample set of following target images in the maintenance site environment is obtained by using a depth camera, and the training sample set of following target images is divided into a training set and a verification set. The convolutional neural network is further trained based on the training set to obtain an initial target recognition model, and the initial target recognition model is verified and optimized based on the verification set.

3. The visual following method of a maintenance vehicle according to claim 1 is characterized in that CSRT target tracking algorithm, specifically including: In a specific maintenance site environment, and when the target is within the specified tracking range, Kalman filtering technology is applied to predict the position and state of the target based on the dynamic model and observation data of the target. During the tracking process, new observation data will be continuously integrated into the target recognition model so that the model can be updated in real time to ensure that it can accurately reflect the evolution of the target's visual features and motion patterns.

4. A control system for a maintenance vehicle, characterized in that Dual closed-loop brushless DC motor speed control system, specifically including: The four motors are controlled by a dual closed-loop DC brushless motor speed control system. The inner loop is the current loop, which feeds back the actual current information through the current acquisition circuit, and the outer loop is the speed measurement loop, which feeds back the actual speed information through the photoelectric encoder.

Images