A method and system for self-following of a target in a complex environment
By combining target detection algorithms and depth information on an automated sweeper, alternative targets are identified and generated, solving the problems of target tracking loss and mistracking in complex environments and achieving more stable target tracking results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN VEHICLE DESIGN & SERVICES CO LTD
- Filing Date
- 2023-02-14
- Publication Date
- 2026-05-29
AI Technical Summary
Existing automated sweepers are prone to target tracking loss and mistracking in complex environments, especially in situations with high pedestrian traffic or when the equipment is obstructed, where the tracking performance of the sensors is unstable.
The algorithm uses target detection to identify the tracking target in the image, uses the target bounding box to obtain the position information, and generates candidate targets by comparing the target bounding box positions in the real-time and previous frame images. It determines whether the tracking target is lost, and if it is lost, the candidate target is used as the new tracking target. The algorithm combines the YOLO target detection algorithm and depth information to improve tracking stability.
It effectively avoids target tracking loss or mistracking in complex environments, significantly improving tracking stability and accuracy.
Smart Images
Figure CN116229416B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and specifically to a method and system for self-following targets in complex environments. Background Technology
[0002] With economic development and rising labor costs, the cleaning of main urban roads has largely become mechanized and automated. Automatic sweepers are commonly used to clean the roads. As a type of sanitation equipment, automatic sweepers integrate road sweeping, garbage collection, and transportation functions. They offer advantages such as saving time and labor, high efficiency, and thorough cleaning, making them widely applicable to cleaning operations on urban streets, industrial parks, sidewalks, and other roads.
[0003] The automated sweeper integrates autonomous sweeping, obstacle avoidance, and intelligent sensing. It primarily relies on sensors added to conventional sweepers to achieve unmanned, automated sweeping, obstacle detection and avoidance of pedestrians or moving objects, and automatic dumping of garbage at designated locations and return to parking spaces after cleaning. Compared to typical manual sweeping equipment, the automated sweeper incorporates cameras, lidar, and ultrasonic sensors, enabling it to perceive and understand its surroundings, achieve real-time scanning and positioning, adapt to various terrains, accurately identify obstacles, and automatically avoid them, achieving fully unmanned sanitation operations.
[0004] Currently, most sensors in automated sweepers use UWB or ultrasonic technology, which are greatly affected by environmental factors. In situations with high pedestrian traffic or obstructed view, target loss is common, resulting in unstable tracking performance. To improve the stability of automated sweeper tracking, some sensors use multi-line LiDAR. Multi-line LiDAR effectively avoids interference from ambient light and strong sunlight, allowing for stable operation both indoors and outdoors. However, it is expensive and prone to target loss and mistracking in complex environments, so tracking stability still needs improvement.
[0005] Therefore, a new environmental perception solution is needed in this field to address the above problems. Summary of the Invention
[0006] The purpose of this invention is to overcome the shortcomings of the prior art described above and to provide a self-following method and system for targets in complex environments.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] A self-following method for targets in complex environments includes:
[0009] The system acquires real-time information on the positional changes of the tracked target in the image, uses a target detection algorithm to identify the tracked target in the image, and uses a bounding box to outline the tracked target.
[0010] Based on the location information of the target bounding box, the speed and steering angle required for vehicle tracking are obtained;
[0011] The position information of the target box in the real-time image is compared with the position information of the target box in the previous frame image to obtain candidate targets;
[0012] Determine whether the target is lost during the tracking process. If it is lost, select a backup target as the tracking target.
[0013] Furthermore, the YOLO object detection algorithm is used to identify the tracking target in the image. When there is one tracking target in the image, the top-left coordinate point of the target bounding box is A1(x1, y1), and the bottom-right coordinate point is B1(x1, y1). When there are multiple tracking targets in the image, the top-left coordinate points of each target bounding box are A1(x1, y1), A2(x2, y2), ..., An(x1, y1), respectively. n y n The coordinates of the lower right corners are B1(x1, y1), B2(x2, y2), ..., Bn(x1, y1), respectively. n y n ).
[0014] Furthermore, the specific method for identifying and tracking targets in an image is as follows:
[0015] Set the number of detection frames for the tracked target to N. frame Calculate a score Min for each tracked target. score ,
[0016] Min score =Dis Mid / Time,
[0017] Among them, Dis Mid =Dis Mid +Dis Now Dis Mid =1 / 2*Width, Time=Time+1, Dis Now =|x n1 -x n2 |;
[0018] Among them, Dis Mid x is the initial distance of the target bounding box from the center of the image, where Width is the width of the image, and x is the distance of the target bounding box from the center of the image. n Here are the coordinates of each bounding box, Time is the number of times the tracked target appears, and Dis is the number of times the target appears. Now This represents the real-time distance of the target bounding box from the center of the image.
[0019] When the number of frames in the image is not greater than the number of detection frames N frame When, the score is Min score The bounding box corresponding to the minimum value is the tracking target.
[0020] Furthermore, when the number of image frames is less than the number of detection frames N frame If no new target is detected in the current image, the score of the currently detected target is calculated.
[0021] If a target is added to the current image, determine whether the target has appeared before. If it has not appeared before, add the target data to the detection array and calculate the score of the new target.
[0022] Filter out the selected target data, and remove Dis Base ≤Dis tol The target data is put into the detection array to calculate the score of the currently detected target, where Dis Base =Hight-y2, Dis Base The distance from the bottom edge of the target bounding box to the bottom edge of the image is given by `Hight`, where `Dist` is the height of the image. tol y1 represents the distance tolerance between the target bounding box and the center of the image, and y2 represents the y-coordinate of the bottom right corner of the current target bounding box.
[0023] Furthermore, when the number of frames in the image equals N frame Then, iterate through the above detection array to find the target score Min. score The target with the minimum value is the tracking target.
[0024] Furthermore, when the number of frames of the image is greater than N frame Real-time calculation of Dis Base Dis Mid The system obtains the position information of the target bounding box and feeds it back to the vehicle controller to obtain the speed and steering angle required for vehicle tracking.
[0025] Furthermore, the specific methods for determining whether the tracked target has been lost during the tracking process are as follows:
[0026] During the tracking process, the difference Dis between the coordinates of a point in the target box in the image and the previous frame is calculated. diff , In the formula, x obj1 x obj2 y obj1 y obj2 Given the coordinates of a point in the bounding box of two images, if Dis diff Less than or equal to Dis tol If so, it is determined that the tracked target has been lost.
[0027] Furthermore, the specific methods for selecting alternative targets are as follows:
[0028] Calculate the difference Dis between the coordinates of the top-left corner of the target bounding box in the real-time image and the coordinates of the top-left corner of the target bounding box in the previous frame image. diff , In the formula, x obj x alter y obj y alter Dis represents the coordinates of the top-left corner of the target bounding box in the two consecutive images. diff The candidate bounding box with the smallest difference;
[0029] Obtain the depth information value of the center position of the real-time target bounding box. obj Depth information value at the center position of the candidate target box alter And calculate the difference between the two, Depth. diff Depth diff =|Depth obj -Depth alter |, when Depth diff With Dis diff At the same time, when the minimum value is reached, the target in the candidate target box becomes the candidate target.
[0030] A self-following system for targets in complex environments, comprising:
[0031] The image acquisition module is used to acquire image information of the tracked target;
[0032] The target recognition module is used to extract the target from the image and outline the target with a bounding box to obtain the position information of the target box.
[0033] The alternative target generation module is used to generate an alternative tracking target in real time based on the position information of the target box during the tracking process;
[0034] The alternative target selection module is used to select an alternative target as the tracking target when the tracking target is lost.
[0035] The follow controller sends tracking commands to the vehicle controller, enabling the vehicle controller to control the speed and steering angle required for the vehicle to follow.
[0036] By adopting the above technical solution, the present invention has the following advantages compared with the prior art:
[0037] This invention identifies the tracking target in an image using a target detection algorithm. Then, it compares the target bounding box position information of the tracking target in the real-time image with the target bounding box position information of the tracking target in the previous frame image to obtain candidate targets. If it is determined that the tracking target is lost during the tracking process, the candidate target is used as the tracking target. This can avoid the situation of target tracking loss or mistracking in crowded, occluded, or complex environments, and greatly improve the stability of tracking. Attached Figure Description
[0038] Figure 1 This is a flowchart of the method in this invention;
[0039] Figure 2 This is a structural block diagram of the system in this invention;
[0040] Figure 3 This is a block diagram illustrating the principle of vehicle following. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0042] Example
[0043] refer to Figure 1 As shown, this invention discloses a self-following method for targets in complex environments, comprising:
[0044] The system acquires real-time information on the positional changes of the tracked target in the image, uses a target detection algorithm to identify the tracked target in the image, and uses a bounding box to outline the tracked target.
[0045] Based on the location information of the target bounding box, the speed and steering angle required for vehicle tracking are obtained;
[0046] The position information of the target box in the real-time image is compared with the position information of the target box in the previous frame image to obtain candidate targets;
[0047] Determine whether the target is lost during the tracking process. If it is lost, select a backup target as the tracking target.
[0048] This invention identifies the tracking target in an image using a target detection algorithm. Then, it compares the bounding box position information of the tracking target in the real-time image with that of the tracking target in the previous frame to obtain candidate targets. If the tracking target is lost during tracking, the candidate target is used as the tracking target. This avoids target tracking loss or mistracking in crowded, occluded, or complex environments, greatly improving tracking stability. The invention will now be described in detail.
[0049] In this embodiment, the YOLO object detection algorithm is used to identify the tracking target in the image. When there is one tracking target in the image, the top-left coordinate point of the target bounding box is A1(x1, y1), and the bottom-right coordinate point is B1(x1, y1). When there are multiple tracking targets in the image, the top-left coordinate points of each target bounding box are A1(x1, y1), A2(x2, y2), ..., An(x1, y1), respectively. n y n The coordinates of the lower right corners are B1(x1, y1), B2(x2, y2), ..., Bn(x1, y1), respectively. n y n ).
[0050] In this embodiment, the specific method for identifying and tracking targets in an image is as follows:
[0051] Set the number of detection frames for the tracked target to N. frame Calculate a score Min for each tracked target. score ,
[0052] Min score =Dis Mid / Time,
[0053] Among them, Dis Mid =Dis Mid +Dis Now Dis Mid =1 / 2*Width, Time=Time+1, Dis Now =|x n1 -x n2 |;
[0054] Among them, Dis Mid The initial distance (Dis) of the target bounding box from the center of the image. Mid In the formula, the value on the left side of the equals sign represents the updated value in the current frame loop, where Width is the width of the image, and x... n Here are the coordinates of each target bounding box, Time is the number of times the tracked target appears (in the Time formula, the left side of the equals sign represents the updated value in the current frame loop), and Dis... Now This represents the real-time distance of the target bounding box from the center of the image.
[0055] When the number of frames in the image is not greater than the number of detection frames N frame When, the score is Min score The bounding box corresponding to the minimum value is the tracking target.
[0056] Furthermore, when the number of image frames is less than the number of detection frames N frameIf no new target is detected in the current image, the score for the currently detected target is calculated. If a target is added to the current image, it is determined whether the target has appeared before. If not, the target data is added to the detection array, and the score for the new target is calculated. To prevent selected targets from being too far from the vehicle, the selected target data is filtered out, and Dis is set to... Base ≤Dis tol The target data is put into the detection array to calculate the score of the currently detected target, where Dis Base =Hight-y2, Dis Base The distance from the bottom edge of the target bounding box to the bottom edge of the image, where High is the height of the image, and Dis is the height of the target bounding box. tol y1 represents the distance tolerance between the target bounding box and the center of the image, and y2 represents the y-coordinate of the bottom right corner of the current target bounding box.
[0057] Furthermore, when the number of frames in the image equals N frame Then, iterate through the above detection array to find the target score Min. score The target with the minimum value is the tracking target.
[0058] Furthermore, when the number of frames of the image is greater than N frame Real-time calculation of Dis Base Dis Mid The system obtains the position information of the target bounding box and feeds it back to the vehicle controller to obtain the speed and steering angle required for vehicle tracking.
[0059] In this embodiment, the specific method for determining whether the tracked target has been lost during the tracking process is as follows:
[0060] During the tracking process, the difference Dis between the coordinates of a point in the target box in the image and the previous frame is calculated. diff , In the formula, x obj1 x obj2 y obj1 y obj2 Given the coordinates of a point in the bounding box of two images, if Dis diff Less than or equal to Dis tol If so, it is determined that the tracked target has been lost.
[0061] In this embodiment, the specific method for selecting candidate targets is as follows:
[0062] Calculate the difference Dis between the coordinates of the top-left corner of the target bounding box in the real-time image and the coordinates of the top-left corner of the target bounding box in the previous frame image. diff , In the formula, x obj x alter y obj y alterDis represents the coordinates of the top-left corner of the target bounding box in the two consecutive images. diff The candidate bounding box with the smallest difference;
[0063] Obtain the depth information value of the center position of the real-time target bounding box. obj Depth information value at the center position of the candidate target box alter And calculate the difference between the two, Depth. diff Depth diff =|Depth obj -Depth alter |, when Depth diff With Dis diff Simultaneously, when the target is at its minimum, the target in the candidate target box is used as the candidate target to prevent tracking loss and mistracking. The real-time acquired image contains the image's depth information; by reading the depth information of each pixel, the depth can be obtained. obj and Depth alter The value of .
[0064] refer to Figure 2 As shown, the present invention also discloses a self-following system for targets in complex environments, comprising:
[0065] The image acquisition module is used to acquire image information of the tracked target;
[0066] The target recognition module is used to extract the target from the image and outline the target with a bounding box to obtain the position information of the target box.
[0067] The alternative target generation module is used to generate an alternative tracking target in real time based on the position information of the target box during the tracking process;
[0068] The alternative target selection module is used to select an alternative target as the tracking target when the tracking target is lost.
[0069] The follow controller sends tracking commands to the vehicle controller, enabling the vehicle controller to control the speed and steering angle required for the vehicle to follow.
[0070] Cooperate Figure 2 and Figure 3 As shown, the vehicle following process is as follows:
[0071] Click the "Follow" button to enter follow mode. Then click "Locate" to initiate the vehicle's follow program. After the program starts, the indicator light will illuminate and the buzzer will sound twice. A person must stand within the sensor 1's detection range and wait 2 seconds. Once the program identifies the target, the follow indicator light will illuminate again. The target maintains a starting distance from the vehicle. When the distance between the target and the vehicle exceeds a threshold, the follow controller calculates the distance difference between them based on the target's position information and the required speed to maintain a stable follow distance. The follow controller sends this speed to the vehicle controller, and the vehicle begins to move forward. During the target's straight-line travel phase, as the distance between the target and the vehicle decreases, the vehicle's follow speed decreases; as the distance increases, the follow speed increases, ensuring the vehicle and target remain stably within the following distance range.
[0072] By filtering the vehicle's speed to smooth it out, the follow controller stops transmitting speed data when the distance between the vehicle and the target is equal to or less than the following threshold. When the target turns, sensor 1 determines the offset between the target and the vehicle's center position. The follow controller calculates the required turning angle based on the target's position information and sends the calculated turning angle to the steering controller to achieve vehicle steering.
[0073] During the following process, the vehicle controller is equipped with sensors 2 on the front and sides of the vehicle to detect the surrounding environment in real time. When an obstacle appears around the vehicle, the vehicle controller calculates the safe distance between the obstacle and the vehicle. When the distance is less than the safe distance, the vehicle controller activates an alarm. The vehicle controller controls the steering of the vehicle, while the follow controller only controls the vehicle speed. When an obstacle appears on the left side of the vehicle and there is no obstacle on the right side, the vehicle controller controls the vehicle to turn right. When an obstacle appears on the right side of the vehicle and there is no obstacle on the left side, the vehicle controller controls the vehicle to turn left. When obstacles appear on both sides of the vehicle and the distance is greater than the minimum passing distance, the vehicle controller controls the vehicle to proceed straight through. When obstacles appear on both sides of the vehicle and the distance is less than the minimum passing distance, the vehicle will stop following. When an obstacle appears directly in front of the vehicle and the vehicle cannot avoid the obstacle by steering, the vehicle will stop following. The follow controller will detect the target in real time. When the target is lost and cannot be detected, the following program will stop, the follow indicator light will turn off, and the buzzer will sound as a warning.
[0074] To prevent vehicles from mistakenly following the target after it is lost, when restarting the self-following program, you need to stand about 60cm directly in front of the vehicle to enter the self-following program; if the target is lost for more than 10 seconds, you need to press the positioning button again and stand about 60cm directly in front of the vehicle to enter the self-following program.
[0075] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A self-following method for targets in complex environments, characterized in that, include: The system acquires real-time information on the positional changes of the tracked target in the image, uses a target detection algorithm to identify the tracked target in the image, and uses a bounding box to outline the tracked target. Based on the location information of the target bounding box, the speed and steering angle required for vehicle tracking are obtained; The position information of the target box in the real-time image is compared with the position information of the target box in the previous frame image to obtain candidate targets; Determine whether the target is lost during the tracking process; if lost, use a backup target as the tracking target. The YOLO object detection algorithm is used to identify tracked targets in an image. When there is one tracked target in the image, the top-left coordinate of the target bounding box is A1(x1, y1), and the bottom-right coordinate is B1(x1, y1). When there are multiple tracked targets in the image, the top-left coordinates of each target bounding box are A1(x1, y1), A2(x2, y2), ..., An(x1, y1), respectively. n y n The coordinates of the lower right corners are B1(x1, y1), B2(x2, y2), ..., Bn(x1, y1), respectively. n y n ); The specific method for selecting candidate targets is as follows: Calculate the difference Dis between the coordinates of the top-left corner of the target bounding box in the real-time image and the coordinates of the top-left corner of the target bounding box in the previous frame image. diff Dis diff = In the formula, x obj x alter y obj y alter Dis represents the coordinates of the top-left corner of the target bounding box in the two consecutive images. diff The candidate bounding box with the smallest difference; Obtain the depth information value of the center position of the real-time target bounding box. obj Depth information value at the center position of the candidate target box alter And calculate the difference between the two, Depth. diff Depth diff =|Depth obj -Depth alter |, when Depth diff With Dis diff At the same time, when the minimum value is reached, the target in the candidate target box becomes the candidate target.
2. The self-following method for a target in a complex environment as described in claim 1, characterized in that, The specific methods for identifying and tracking targets in an image are as follows: Set the number of detection frames for the tracked target to N. frame Calculate a score Min for each tracked target. score , My score =Haze Mid / Hour, Among them, Dis Mid = Dis Mid + Dis Now , Dis Mid = 1 / 2 * Width, Time = Time + 1, Dis Now = |x n1 - x n2 |; Among them, Dis Mid x is the initial distance of the target bounding box from the center of the image, where Width is the width of the image, and x is the distance of the target bounding box from the center of the image. n Here are the coordinates of each bounding box, Time is the number of times the tracked target appears, and Dis is the number of times the target appears. Now This represents the real-time distance of the target bounding box from the center of the image. When the number of frames in the image is not greater than the number of detection frames N frame When, the score is Min score The bounding box corresponding to the minimum value is the tracking target.
3. The self-following method for a target in a complex environment as described in claim 2, characterized in that, When the number of frames in the image is less than the number of detection frames N frame If no new target is detected in the current image, the score of the currently detected target is calculated. If a target is added to the current image, determine whether the target has appeared before. If it has not appeared before, add the target data to the detection array and calculate the score of the new target. Filter out the selected target data, and remove Dis Base ≤Dis tol The target data is put into the detection array to calculate the score of the currently detected target, where Dis Base =Hight-y2, Dis Base The distance from the bottom edge of the target bounding box to the bottom edge of the image is given by `Hight`, where `Dist` is the height of the image. tol y1 represents the distance tolerance between the target bounding box and the center of the image, and y2 represents the y-coordinate of the bottom right corner of the current target bounding box.
4. The self-following method for a target in a complex environment as described in claim 3, characterized in that, When the number of frames of the image equals N frame Then, iterate through the above detection array to find the target score Min. score The target with the minimum value is the tracking target.
5. The self-following method for a target in a complex environment as described in claim 3, characterized in that, When the number of frames of the image is greater than N frame Real-time calculation of Dis Base Dis Mid The system obtains the position information of the target bounding box and feeds it back to the vehicle controller to obtain the speed and steering angle required for vehicle tracking.
6. The self-following method for a target in a complex environment as described in claim 3, characterized in that, The specific methods for determining whether the tracked target has been lost during the tracking process are as follows: During the tracking process, the difference Dis between the coordinates of a point in the target box in the image and the previous frame is calculated. diff Dis diff = In the formula, x obj1 x obj2 y obj1 y obj2 Given the coordinates of a point in the bounding box of two images, if Dis diff Less than or equal to Dis tol If so, it is determined that the tracked target has been lost.
7. A self-following system for a target in a complex environment, used to perform the method as described in any one of claims 1-6, characterized in that, include: The image acquisition module is used to acquire image information of the tracked target; The target recognition module is used to extract the target from the image and outline the target with a bounding box to obtain the position information of the target box. The alternative target generation module is used to generate an alternative tracking target in real time based on the position information of the target box during the tracking process; The alternative target selection module is used to select an alternative target as the tracking target when the tracking target is lost. The follow controller sends tracking commands to the vehicle controller, enabling the vehicle controller to control the speed and steering angle required for the vehicle to follow.