Target tracking method based on environmental perception

Abstract

The invention relates to a target tracking method based on environmental perception, and belongs to the field of artificial intelligence and target tracking, and the target tracking method continuously monitors the motion state of a target object through the combination of target tracking and semantic segmentation, provides support for decision making, and improves the target tracking efficiency. Comprising the steps of image acquisition and preprocessing, environmental perception semantic segmentation, boundary fitting post-processing, target tracking and tracking parameter automatic updating. The target tracking method based on environmental perception is characterized in that semantic segmentation can provide in-depth understanding of a scene, including position, state and context information of a target, support is provided for decision making, and through combination of target tracking and semantic segmentation, mutual complementation can be achieved, the adaptability and robustness of a system to a complex environment are improved, and the target tracking efficiency is improved. And a solid foundation is provided for the development of an intelligent system. And target tracking parameters are automatically updated, so that stable cross-scene tracking is ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology and target tracking technology, and in particular to a target tracking method based on environment perception. Background Art

[0002] Target tracking technology is to identify a specific target from a video frame, and then continuously track the position and movement of the target in subsequent frames. It plays an important role in multiple application scenarios such as intelligent monitoring and autonomous driving. Traditional target tracking methods include feature-based methods and correlation-based methods. Feature-based methods match by extracting feature points of the target, but are easily affected by lighting changes and occlusions. Correlation-based methods locate the target by calculating the correlation of the target template in the search area, but the generalization ability of the model is limited and it is easily disturbed when switching between different scenes, resulting in target loss. Summary of the invention

[0003] To solve the above technical problems, the present invention provides a target tracking method based on environment perception, which continuously monitors the motion state of the target object by combining target tracking and semantic segmentation to provide support for decision-making, and includes the following steps:

[0004] Step S1: using the collected image of the optical lens and camera, evaluating the image average brightness and variance information, determining whether the current image illumination is appropriate, and completing the image preprocessing;

[0005] Step S2: Segment the image pixel by pixel using a semantic segmentation algorithm to determine the environment area and obtain a binary image;

[0006] Step S3: using a morphologically based expansion and erosion algorithm to obtain the boundary areas of different environmental areas and fit the boundary lines between different areas;

[0007] Step S4: Use the target tracking algorithm to track the object in the image and obtain the coordinate information of the object in each frame;

[0008] Step S5: determine the intersection relationship between the target coordinates and the environment binary map to obtain the environment area of ​​the current target;

[0009] Step S6: Calculate the distance between the target coordinates and the boundary line, and determine whether the current target is in a cross-region state in the boundary line area;

[0010] Step S7: According to the target area and state obtained in steps S5 and S6, the KCF tracking algorithm automatically updates parameters to adapt to stable tracking in different scenarios;

[0011] In step S2, the captured image is segmented pixel by pixel using the semantic segmentation algorithm Enet: Enet (Efficient Neural Network) is an efficient neural network for semantic segmentation, which aims to achieve high-precision image semantic segmentation. The input image resolution ranges from 416*416*3 to 1024*1024*3 to adapt to different computing platforms, including HiSilicon HI3559A, Rockchip RK3588 and server-side operating environment. The model output is 416*416*4 binary images of environmental areas such as sky, water surface, buildings and background. The model structure of Enet has 20 layers, mainly including convolutional layer bottleneck structure, downsampling and upsampling modules. The number of output channels increases from 16 to 64 and then to 128, and finally reduces to 4. The hole convolution part uses hole coefficients of 2, 4, 8, and 16 respectively.

[0012] In one embodiment of the present invention, step S3 further includes the following sub-steps:

[0013] S301: For the environment areas such as the sky, water surface, and land buildings, the three binary images are respectively subjected to morphological dilation operations using 3*3 convolution kernels to expand the boundary areas and fill the internal holes;

[0014] S302: Obtain the intersection area of ​​the binary images between the sky and the water surface, the sky and the land, and the water surface and the land;

[0015] S303: Obtaining contour constituent points for the intersection area, and using the least squares method to fit the boundary lines between two different areas, including the sky-water boundary line, the sky-land boundary line, and the land-water boundary line.

[0016] In one embodiment of the present invention, the intersection relationship of step S5 is specifically:

[0017] If the tracking target coordinates are within the range of the sky binary map, flag1=1; if the tracking target coordinates are within the range of the water surface binary map, flag1=2; if the tracking target coordinates are within the range of the ground building binary map, flag1=3.

[0018] In one embodiment of the present invention, determining the relationship between the tracking target coordinates and the boundary lines between multiple environments in step S6 includes the following sub-steps:

[0019] S601: Obtain the center coordinates and width and height of the tracking target frame;

[0020] S602: Calculate the Euclidean distance between the center coordinate of the target frame and every two points of the environment boundary line to obtain the minimum distance d between the center coordinate and the boundary line;

[0021] S603: If the minimum distance d is greater than the target width and height, it is considered that the current target is in a single environment and flag2=0; if the minimum distance d is less than or equal to the target width and height, it is considered that the target intersects with the environment boundary and is moving across regions; if the target intersects with the sky-water boundary, flag2=1; if the target intersects with the land-water boundary, flag2=2; if the target intersects with the sky-land boundary, flag2=3.

[0022] In one embodiment of the present invention, in step S7, the KCF tracking algorithm determines the adaptive parameter update according to the environmental boundary line, including the following sub-steps:

[0023] S701: according to the determination in step S6 that the target has reached the boundary, local contour enhancement in the search area is enabled to prevent the target from being similar to the background and thus drowning the target;

[0024] S702: Reduce the tracking search area and reduce the update rate of the target template;

[0025] S703: Increase the APCE value to stop updating the target template, implement memory tracking, and prevent the target template from being contaminated in the boundary area.

[0026] The above technical solution of the present invention has the following advantages compared with the prior art:

[0027] 1. Compared with traditional target tracking solutions, this system solution combines semantic segmentation algorithm and target tracking algorithm to perceive the surrounding environment in real time, and can adapt to stable tracking in various environments.

[0028] 2. Initialize the target tracking process through the category and boundary information of semantic segmentation, and provide auxiliary information during the tracking process to cope with complex situations such as occlusion, lighting and environmental changes;

[0029] The semantic segmentation technology enhances the machine's perception of the surrounding environment by identifying the category of each pixel in the image, such as distinguishing the sky, water surface, buildings, etc.

[0030] By combining target tracking and semantic segmentation, they can complement each other, improve the system's adaptability and robustness to complex environments, and provide a solid foundation for the development of intelligent systems.

[0031] 3. In practical applications, the automatic parameter update module can automatically adjust the parameters of the tracking algorithm according to changes in the target's environment, avoiding the complexity of manual adjustment and taking into account the intelligence and flexibility of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0033] Figure 1a , 1b 1c is the detection effect of different environment perception boundary lines provided by the present invention;

[0034] Figure 2a , 2b 2c is the stable tracking effect of the target tracking algorithm provided by the present invention between different scenes;

[0035] Figure 3 It is an algorithm flow chart of the target tracking method based on environment perception provided by the present invention. DETAILED DESCRIPTION

[0036] like Figure 3 As shown, this embodiment provides a target tracking method based on environment perception, which continuously monitors the motion state of the target object by combining target tracking and semantic segmentation to provide support for decision-making; the method includes the following steps:

[0037] Step S1: using the collected image of the optical lens and camera, evaluating the image average brightness and variance information, determining whether the current image illumination is appropriate, and completing the image preprocessing;

[0038] Step S2: Segment the image pixel by pixel using a semantic segmentation algorithm to determine the environment area and obtain a binary image;

[0039] Step S3: using a morphologically based expansion and erosion algorithm to obtain the boundary areas of different environmental areas and fit the boundary lines between different areas;

[0040] Step S4: Use the target tracking algorithm to track the object in the image and obtain the coordinate information of the object in each frame;

[0041] Step S5: determine the intersection relationship between the target coordinates and the environment binary map to obtain the environment area of ​​the current target;

[0042] Step S6: Calculate the distance between the target coordinates and the boundary line, and determine whether the current target is in a cross-region state in the boundary line area;

[0043] Step S7: According to the target area and state obtained in steps S5 and S6, the KCF tracking algorithm automatically updates parameters to adapt to stable tracking in different scenarios.

[0044] At the same time Figure 1a , 1bAs shown in Figure 1c, by combining object tracking and semantic segmentation algorithms, a more powerful and intelligent system is formed. Semantic segmentation provides a deep understanding of the scene, including the location, category, and contextual information of the object. Object tracking can continuously monitor the motion status of these objects to support decision making.

[0045] Step S2 uses the semantic segmentation algorithm Enet to segment the captured image pixel by pixel. The Enet network input is 416*416*3, and after forward reasoning, it outputs a 416*416*4 result, which are binary image masks of the sky, water surface, building, and background areas.

[0046] Specifically, semantic segmentation is a very challenging and widely used technology in the field of computer vision. Its core task is to identify the semantic information of each area in the image through pixel-level classification. This technology can not only identify objects in the image, but also accurately depict the outline of the object, thereby achieving a deep understanding of the image structure. Enet (Efficient Neural Network), as a leader in this field, is an efficient neural network model designed specifically for semantic segmentation tasks. The original intention of Enet's design is to achieve high-precision image semantic segmentation while maintaining a balance between the model's computational efficiency and resource consumption.

[0047] Enet uses a typical encoder-decoder structure, which enables the model to extract features of the image in the encoding stage and then map these features back to the semantic segmentation results of the original resolution in the decoding stage. The network structure of Enet is carefully designed, including four main parts: the initial block, encoder, decoder, and final block. The initial block is the starting point of the model and is responsible for preprocessing and preliminary feature extraction of the input image. This part usually includes a series of convolutional layers and normalization layers, such as batch normalization, which help extract basic low-level features such as edges and textures from the original image. Next is the encoder part, which gradually reduces the size and number of channels of the feature map by stacking convolutional layers and pooling operations, while increasing the depth of the features to extract high-level features of the image. The design of this part is very critical because it directly affects the model's ability to understand the content of the image. In order to further improve the efficiency of the model, Enet uses Depthwise Separable Convolution in Encoder, which can significantly reduce the number of parameters and computational complexity of the model without sacrificing the ability to extract features. The Decoder part is another core of Enet. It restores the resolution of the segmentation result through upsampling and transposed convolution operations and maps the low-dimensional features back to the original image size. Upsampling increases the size of the feature map through interpolation technology, while transposed convolution is used to further refine the features on the basis of upsampling to achieve fine mapping of features. This process not only requires accurate restoration of the spatial resolution of the image, but also requires maintaining the semantic information of the features, which is crucial for the final semantic segmentation results. The final Block is the key step for Enet to generate the final semantic segmentation map. This part may contain multiple convolutional layers to further refine the features and generate the final segmentation map. In addition, the final Block will also contain activation functions, such as Softmax or Sigmoid, which are used to convert the output of the convolution layer into a probability distribution, thereby achieving pixel-level category prediction. These operations ensure that the final output semantic segmentation map is highly accurate and readable.

[0048] To further improve the performance of Enet, the model also adopts a series of optimization strategies. For example, dilated convolution can increase the receptive field without increasing the number of parameters by introducing holes in the convolution kernel, which helps to extract a wider range of contextual information. 1x1 convolution is used to reorganize information between feature channels, which helps to extract more effective feature representations. Parallel branches can capture local and global information of the image at the same time by processing feature maps of different resolutions in parallel, enhancing the generalization ability of the model. Feature fusion can improve the model's ability to capture details by fusing high-level features in the encoder with low-level features in the decoder. These optimization strategies enable Enet to achieve fast and accurate semantic segmentation on resource-constrained devices, suitable for various real-time image analysis tasks.

[0049] Step S3 uses a morphologically based expansion and erosion algorithm to obtain the boundary area of ​​different environmental areas for the binary image. This process can be achieved by the following steps:

[0050] (1) Use a 3*3 convolution kernel to perform morphological dilation on the three binary images of sky, water surface, and land buildings, expand the boundary area and fill the internal holes. (2) Calculate the intersection area between the sky, water surface, sky, land, and water surface and land. (3) Calculate the contour points of the intersection area and use the least squares method to fit the boundary lines between different areas.

[0051] Step S4 uses the target tracking algorithm KCF to track the object in the captured image. KCF (Kernelized Correlation Filters) is a classic algorithm for target tracking, and its principle is based on kernel correlation filters. Target tracking is achieved by mapping the target template to a high-dimensional feature space and using filters to evaluate the similarity between the target template and the candidate region. Specifically, the KCF algorithm includes four main steps: feature extraction, kernel function mapping, filter training, and target tracking. In the feature extraction stage, features are first extracted from the target template and the candidate region, and techniques such as fast Fourier transform (FFT) are usually used to accelerate the calculation. Then, these features are mapped to a high-dimensional feature space using a kernel function to better distinguish the target from the background. Next, in the filter training stage, a set of kernel correlation filters are obtained by learning the linear relationship between the target template and the candidate region. Finally, in the target tracking stage, the trained filter is used to track the new image frame, that is, the region most similar to the target template is selected in the candidate region as the position of the target. The KCF algorithm is fast and accurate and is suitable for real-time target tracking tasks.

[0052] Step S5 determines the intersection relationship between the tracking target coordinates and the environment binary map to obtain the environment area where the current target is located. Specifically: if the tracking target coordinates are within the sky binary map range, flag1 = 1; if the tracking target coordinates are within the water surface binary map range, flag1 = 2; if the tracking target coordinates are within the ground building binary map range, flag1 = 3.

[0053] Step S6 determines the relationship between the tracking target coordinates and the dividing lines between multiple environments. Specifically, it includes: (1) obtaining the center coordinates and width and height of the tracking target frame; (2) calculating the Euclidean distance between the center coordinates of the target frame and every two points on the environmental dividing line, and obtaining the minimum distance d between the center coordinates and the dividing line; (3) If the minimum distance d is greater than the target width and height, it is considered that the current target is in a single environment and flag2=0; if the minimum distance d is less than or equal to the target width and height, it is considered that the target intersects with the environmental dividing line and is moving across regions. If the target intersects with the sky-water dividing line, flag2=1; if the target intersects with the land-water dividing line, flag2=2; if the target intersects with the sky-land dividing line, flag2=3. Among them, Figure 2a , 2b , as shown in 2c.

[0054] Step S7 adaptively updates the parameters based on the judgment of the environmental boundary line in S6. Specifically, it includes: (1) based on the judgment of S6 that the target has reached the boundary line, the local contour enhancement of the search area is turned on to prevent the target from being similar to the background and drowning the target; (2) the tracking search area is reduced and the update rate of the target template is reduced;

[0055] (3) Increase the APCE value to stop updating the target template, implement memory tracking, and prevent the target template from being contaminated in the boundary area.

[0056] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A target tracking method based on environment perception, wherein the target tracking method continuously monitors the motion state of the target object by combining target tracking and semantic segmentation to provide support for decision making, characterized in that: The steps include: Step S1: using the collected image of the optical lens and camera, evaluating the image average brightness and variance information, determining whether the current image illumination is appropriate, and completing the image preprocessing; Step S2: Segment the image pixel by pixel using a semantic segmentation algorithm to determine the environment area and obtain a binary image; Step S3: using a morphologically based expansion and erosion algorithm to obtain the boundary areas of different environmental areas and fit the boundary lines between different areas; Step S4: Use the target tracking algorithm to track the object in the image and obtain the coordinate information of the object in each frame; Step S5: determine the intersection relationship between the target coordinates and the environment binary map to obtain the environment area of ​​the current target; Step S6: Calculate the distance between the target coordinates and the boundary line, and determine whether the current target is in a cross-region state in the boundary line area; Step S7: According to the target area and state obtained in steps S5 and S6, the KCF tracking algorithm automatically updates parameters to adapt to stable tracking in different scenarios; In step S2, the captured image is segmented pixel by pixel using the semantic segmentation algorithm Enet; and the input image resolution ranges from 416*416*3 to 1024*1024*3 to adapt to different computing platforms. At the same time, the model output is a 416*416*4 binary image of the environmental area including the sky, water surface, building and background respectively; the model structure of Enet has 20 layers, mainly including a convolutional layer bottleneck structure, downsampling and upsampling modules, and the number of output channels increases from 16 to 64 and then to 128, and finally reduces to 4; the atrous convolution part uses atrous coefficients of 2, 4, 8, and 16 respectively.

2. The target tracking method according to claim 1, characterized in that: Step S3 also includes the following sub-steps: S301: For the environment areas such as the sky, water surface, and land buildings, the three binary images are respectively subjected to morphological dilation operations using 3*3 convolution kernels to expand the boundary areas and fill the internal holes; S302: Obtain the intersection area of ​​the binary images between the sky and the water surface, the sky and the land, and the water surface and the land; S303: Obtaining contour constituent points for the intersection area, and using the least squares method to fit the boundary lines between two different areas, including the sky-water boundary line, the sky-land boundary line, and the land-water boundary line.

3. The target tracking method according to claim 1, characterized in that: The intersection relationship of step S5 is specifically: If the tracking target coordinates are within the range of the sky binary map, flag1=1; if the tracking target coordinates are within the range of the water surface binary map, flag1=2; if the tracking target coordinates are within the range of the ground building binary map, flag1=3.

4. The target tracking method according to claim 1, characterized in that: Determining the relationship between the tracking target coordinates and the boundary lines between multiple environments in step S6 includes the following sub-steps: S601: Obtain the center coordinates and width and height of the tracking target frame; S602: Calculate the Euclidean distance between the center coordinate of the target frame and every two points of the environment boundary line to obtain the minimum distance d between the center coordinate and the boundary line; S603: If the minimum distance d is greater than the target width and height, it is considered that the current target is in a single environment and flag2=0; if the minimum distance d is less than or equal to the target width and height, it is considered that the target intersects with the environment boundary and is moving across regions; if the target intersects with the sky-water boundary, flag2=1; if the target intersects with the land-water boundary, flag2=2; if the target intersects with the sky-land boundary, flag2=3.

5. The target tracking method according to claim 1, characterized in that: In step S7, the KCF tracking algorithm determines the adaptive parameter update according to the environmental boundary line, including the following sub-steps: S701: according to the determination in step S6 that the target has reached the boundary, local contour enhancement in the search area is enabled to prevent the target from being similar to the background and thus drowning the target; S702: Reduce the tracking search area and reduce the update rate of the target template; S703: Increase the APCE value to stop updating the target template, implement memory tracking, and prevent the target template from being contaminated in the boundary area.

Images