Target tracking method and device, electronic device and medium

Abstract

The present disclosure discloses a target tracking method and device, electronic equipment and storage medium. The target tracking method comprises: based on the gray value of the pixels in the target image, performing intensity layering on the gray value to obtain one or more gray scale images; performing multi-scale structure feature extraction on the gray scale images to obtain multi-scale structure features; and performing target tracking according to the multi-scale structure features.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a target tracking method and apparatus, electronic device and storage medium. Background Technology

[0002] Target tracking has wide applications in fields such as security and / or autonomous driving.

[0003] Target tracking can be based on acquired images.

[0004] In related technologies, tracking algorithms based on partial image regions within an image often cannot handle complex image scenes, and cannot achieve target detection in videos with severe occlusion; feature-based tracking methods often fail to distinguish the features of multiple similar targets when there are many targets in the video image, leading to failure.

[0005] Model-based tracking methods require extensive training to build the model beforehand. When the target changes, the original model becomes unusable, resulting in narrow applicability. Detection-based tracking algorithms, on the other hand, need to constantly update their detectors, leading to high resource consumption and low operating efficiency. Summary of the Invention

[0006] This disclosure provides a target tracking method, apparatus, electronic device, and storage medium to reduce the target tracking loss rate without requiring extensive data model training.

[0007] The first aspect of this disclosure provides a target tracking method, including:

[0008] Based on the grayscale values ​​of pixels within the target image, intensity layering of the grayscale values ​​is performed to obtain one or more grayscale images;

[0009] Multi-scale structural features are extracted from the grayscale image to obtain multi-scale structural features.

[0010] Target tracking is performed based on the multi-scale structural features.

[0011] Based on the above scheme, the target tracking based on the multi-scale structural features includes:

[0012] The candidate region of the nth target in the m+1th frame image based on the mth frame image; where m and n are both positive integers;

[0013] Determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the (m+1)-th frame image;

[0014] Based on the similarity, the position of the nth target in the (m+1)th frame of the image is determined.

[0015] Based on the above scheme, the candidate region of the nth target in the m+1th frame image based on the mth frame image includes at least one of the following:

[0016] Based on the optical flow estimation of the nth target in the mth frame image, the candidate region of the mth target in the (m+1)th frame image is determined;

[0017] The (m+1)th frame image is divided into multiple image regions based on the s-th sliding window, and the candidate region is determined based on the pixel similarity between the image region and the region where the nth target is located in the m-th frame image; wherein, s is a positive integer less than or equal to S; S is a positive integer not less than 2; and the size of different sliding windows is different.

[0018] Based on the above scheme, the step of obtaining one or more grayscale images by intensity layering of the grayscale values ​​of pixels within the target image includes:

[0019] Intensity-based layering is performed on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images.

[0020] Wherein, Pm is the number of local minima of the grayscale value of the pixel in the target image, and the grayscale values ​​of pixels in different grayscale images are located between different local minima groups. A local minima group includes two adjacent local minima.

[0021] Based on the above scheme, the step of performing intensity layering based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images includes:

[0022] Based on the grayscale values ​​of the pixels, the target image is divided into multiple layered images; different layered images contain pixels with different grayscale values.

[0023] Determine the number of local minima contained in pixels of two adjacent layered images;

[0024] When the number of local minima is equal to a preset value, the two adjacent layered images are respectively determined as grayscale images.

[0025] Based on the grayscale values ​​of pixels within the target image, intensity layering of the grayscale values ​​is performed to obtain one or more grayscale images.

[0026] Based on the above scheme, the step of performing intensity layering based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images includes:

[0027] When the number of local minima is less than the preset value, a layered image is inserted between two adjacent layered images;

[0028] And / or,

[0029] When the number of local minima is greater than the preset value, the two adjacent layered images are merged.

[0030] Based on the above scheme, the method further includes:

[0031] The image to be processed is subjected to skip convolution to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

[0032] Based on the above scheme, the method further includes:

[0033] The image region of the original image is divided into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0034] The second type of region of the original image is subjected to target enhancement processing to obtain the image to be processed.

[0035] Based on the above scheme, the step of extracting multi-scale structural features from the grayscale image to obtain multi-scale structural features includes:

[0036] Each of the grayscale images is processed based on grayscale values ​​to obtain one or more of the following: multidimensional grayscale features, multidimensional grayscale variation features, multidimensional effective patch ratio, patch edge variation features, and target scale distribution features.

[0037] A second aspect of this disclosure provides a target tracking method, including:

[0038] The target image is divided into a first type of region and a second type of region; wherein the difference between the target and the background in the first type of region is greater than the difference between the target and the background in the second type of region;

[0039] The second type of region of the target image is subjected to target enhancement processing to obtain the image to be processed;

[0040] Target tracking is performed based on the image to be processed.

[0041] Based on the above scheme, the step of performing target enhancement processing on the second type of region of the target image to obtain the image to be processed includes:

[0042] A high-hat transformation is performed on the second type of region of the target image to obtain the image to be processed for target enhancement.

[0043] A third aspect of this disclosure provides a target tracking device, comprising:

[0044] A layering module is used to perform intensity layering of gray values ​​based on the gray values ​​of pixels within a target image to obtain one or more grayscale images.

[0045] The extraction module is used to extract multi-scale structural features from the grayscale image to obtain multi-scale structural features;

[0046] The tracking module is used to track the target based on the multi-scale structural features.

[0047] Based on the above scheme, the tracking module is specifically used to determine the candidate region of the nth target in the m+1th frame image based on the m-th frame image; wherein m and n are both positive integers; determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the m+1th frame image; and determine the position of the nth target in the m+1th frame image based on the similarity.

[0048] Based on the above scheme, the tracking module is specifically used to perform at least one of the following:

[0049] Based on the optical flow estimation of the nth target in the mth frame image, the candidate region of the mth target in the (m+1)th frame image is determined;

[0050] The (m+1)th frame image is divided into multiple image regions based on the s-th sliding window, and the candidate region is determined based on the pixel similarity between the image region and the region where the nth target is located in the m-th frame image; wherein, s is a positive integer less than or equal to S; S is a positive integer not less than 2; and the size of different sliding windows is different.

[0051] Based on the above scheme, the layering module is specifically used to perform intensity layering according to the gray values ​​of the pixels of the target image to obtain Pm-1 grayscale images, wherein Pm is the number of local minima of the gray values ​​of the pixels of the target image, and the gray values ​​of the pixels of different grayscale images are located between different local minima groups. A local minima group includes two adjacent local minima.

[0052] Based on the above scheme, the layering module is further specifically used to divide the target image into multiple layered images according to the grayscale value of the pixel; different layered images contain pixels with different grayscale values; determine the number of local minima contained in the pixels of two adjacent layered images; when the number of local minima is equal to a preset value, the two adjacent layered images are respectively determined as the grayscale images.

[0053] Based on the above scheme, the layering module is specifically used to insert a layered image into two adjacent layered images when the number of local minima is less than the preset value; and / or to merge two adjacent layered images when the number of local minima is greater than the preset value.

[0054] Based on the above scheme, the device further includes:

[0055] The skip convolution module is used to perform skip convolution processing on the image to be processed to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

[0056] Based on the above scheme, the device further includes:

[0057] The segmentation module is used to divide the image region of the original image into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0058] An enhancement module is used to perform target enhancement processing on the second type of region of the original image to obtain the image to be processed.

[0059] Based on the above scheme, the extraction module is specifically used to process each grayscale image based on grayscale value to obtain one or more of the following: multidimensional grayscale features, multidimensional grayscale change features, multidimensional effective patch ratio, patch edge change features, and target scale distribution features.

[0060] A fourth aspect of this disclosure provides a target tracking device, comprising:

[0061] The region module is used to divide the image region of the target image into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0062] The image to be processed module is used to perform target enhancement processing on the second type of region of the target image to obtain the image to be processed;

[0063] The target module is used for target tracking based on the image to be processed.

[0064] Based on the above scheme, the image to be processed module is specifically used to perform a high-hat transformation on the second type of region of the original image to obtain the image to be processed for target enhancement processing.

[0065] A fifth aspect of this disclosure provides an electronic device, comprising:

[0066] Memory;

[0067] A processor, connected to the memory, is configured to implement the target tracking method provided by any of the foregoing first and / or second aspects by executing computer-executable instructions stored in the memory.

[0068] A sixth aspect of this disclosure provides a computer storage medium, characterized in that the computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, they can realize the target tracking method provided by any of the aforementioned first and / or second aspects.

[0069] The technical solution provided in this disclosure, which performs intensity layering based on the grayscale value of pixels, will obtain multiple grayscale images. More multi-dimensional multi-scale structural features can be easily extracted from the grayscale images. Thus, accurate target tracking can be achieved without using more sample data to train the model, and the phenomenon of target loss is less.

[0070] The technical solution provided in this disclosure divides the target image into regions, enhances regions where the target and background are not significantly different, and obtains the image to be processed. Then, target tracking is performed on the image to be processed, reducing the target loss phenomenon caused by background interference, and without the need to add additional training data for the tracking model. Attached Figure Description

[0071] Figure 1 A flowchart illustrating a target tracking method provided in an embodiment of this disclosure;

[0072] Figure 2 A flowchart illustrating a target tracking method provided in an embodiment of this disclosure;

[0073] Figure 3 A flowchart illustrating a target tracking method provided in an embodiment of this disclosure;

[0074] Figure 4 A flowchart illustrating a target tracking method provided in an embodiment of this disclosure;

[0075] Figure 5 This is a schematic diagram of the structure of a video acquisition and encoding system for target tracking provided in an embodiment of the present disclosure;

[0076] Figure 6 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of the present disclosure;

[0077] Figure 7 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of this disclosure;

[0078] Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0079] To gain a more detailed understanding of the features and technical content of this disclosure, the implementation of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit this disclosure.

[0080] like Figure 1 As shown, this disclosure provides a target tracking method, including:

[0081] S110: Based on the gray values ​​of pixels in the target image, perform intensity layering of the gray values ​​to obtain one or more grayscale images;

[0082] S120: Perform multi-scale structural feature extraction on the grayscale image to obtain multi-scale structural features;

[0083] S130: Perform target tracking based on the multi-scale structural features.

[0084] The target tracking method provided in this disclosure can be applied to various electronic devices, including but not limited to: terminals and / or servers.

[0085] The terminal includes, but is not limited to, image acquisition devices, such as surveillance equipment along roadsides.

[0086] The servers mentioned include, but are not limited to, monitoring servers.

[0087] In this embodiment of the disclosure, after the electronic device receives a target image to be processed, it performs intensity layering of the grayscale values ​​based on the grayscale values ​​of each pixel in the target image. Different grayscale values ​​correspond to different intensities. This results in a grayscale image containing all the included grayscale values.

[0088] When extracting features, multi-scale structural features are extracted from the grayscale images separately. In this way, converting a target image into multiple grayscale images, compared to directly binarizing the target image to obtain a single image for feature map extraction, retains more features of the tracked target, thus obtaining multi-scale structural features.

[0089] Then, target tracking is performed based on these multi-scale structural features, which allows for more reference features, thus reducing the phenomenon of target loss, and without increasing the training data of the model.

[0090] For example, in some embodiments, various types of machine learning models are used to generate the grayscale image; and features of the grayscale image are extracted to obtain multi-scale structural features; finally, the machine learning model performs multi-scale structural feature extraction to determine which region of the image the target is in or not in.

[0091] In this embodiment of the disclosure, the target tracking can be single-target tracking or multi-target tracking.

[0092] If the target image contains multiple targets to be tracked, multi-target tracking is performed; if it contains only one target, single-target tracking is performed.

[0093] like Figure 2 As shown, S130 may include:

[0094] S131: The candidate region of the nth target in the m+1th frame image based on the mth frame image; where m and n are both positive integers;

[0095] S132: Determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the (m+1)-th frame image;

[0096] S133: Determine the position of the nth target in the (m+1)th frame image based on the similarity.

[0097] The candidate region can be the potential region of the nth target in the (m+1)th frame of the image.

[0098] In this embodiment of the disclosure, multi-scale structural features of each candidate region are extracted using steps S110 to S120, and similarity is calculated between the multi-scale structural features of the nth target in the mth frame image, thereby determining which candidate region is the region where the nth target is located in the (m+1)th frame image based on the similarity.

[0099] For example, the candidate region with a similarity greater than a preset threshold and the maximum similarity is determined as the position of the nth target in the (m+1)th frame image.

[0100] S131 may include at least one of the following:

[0101] Based on the optical flow estimation of the nth target in the mth frame image, the candidate region of the mth target in the (m+1)th frame image is determined;

[0102] The (m+1)th frame image is divided into multiple image regions based on the s-th sliding window, and the candidate region is determined based on the pixel similarity between the image region and the region where the nth target is located in the m-th frame image; wherein, s is a positive integer less than or equal to S; S is a positive integer not less than 2; and the size of different sliding windows is different.

[0103] Based on the optical flow characteristics obtained from optical flow estimation, a candidate region will be obtained, and the nth target may be within this first type of candidate region.

[0104] At the same time, a sliding window is set up. The sliding window is moved to traverse the target image. Then, based on the number of overlaps between the pixel values ​​of the pixels contained in the sliding window and the pixel values ​​of the nth target image region in the nth frame, it is determined whether the region is a candidate region. If the number of overlaps is greater than a preset value, it can be considered a candidate region; otherwise, it can be considered not a candidate region.

[0105] In addition, since the tracked target may move closer to or further away from the image acquisition device during its movement, the number of pixels occupied in the m-th frame and the (m+1)-th frame is different.

[0106] Therefore, in order to comprehensively locate the candidate regions, the window size will be varied to traverse the target image.

[0107] For example, the size of the initial sliding window is determined according to the image size of the nth target in the mth frame. Then, the initial window is scaled according to a certain scaling ratio to obtain sliding windows for different scan images.

[0108] The candidate regions obtained based on the sliding window can be classified as the second type of candidate regions.

[0109] Since both the first and second candidate regions will be processed through steps S110 to S120, the phenomenon of target loss due to incorrect candidate region selection can be reduced, thereby improving the target tracking success rate, and no additional sample data is required to train the model.

[0110] In this embodiment of the disclosure, step S110 may include:

[0111] Intensity-based layering is performed on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images.

[0112] Wherein, Pm is the number of local minima of the grayscale value of the pixel in the target image, and the grayscale values ​​of pixels in different grayscale images are located between different local minima groups. A local minima group includes two adjacent local minima.

[0113] For example, by performing histogram statistics on the grayscale values ​​of all pixels in the target image, local minima of the grayscale values ​​of pixels in the target image can be found. Using the local minima as the dividing point, pixels with grayscale values ​​located between two adjacent local minima are grouped into a grayscale map, while pixels with grayscale values ​​not located between two adjacent local minima have a pixel value of 0 in this grayscale map.

[0114] In this way, the target image is segmented according to its gray value distribution, which is based on its own gray value distribution.

[0115] In one embodiment, when performing grayscale image layering, the grayscale value distribution of the target image can be statistically analyzed before layering.

[0116] In other embodiments, the layering can be performed by a machine learning model such as a neural network, and the intensity layering of the grayscale values ​​of the target image by the machine learning model can be performed in the following manner.

[0117] The step of performing intensity layering based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images includes:

[0118] Based on the grayscale values ​​of the pixels, the target image is divided into multiple layered images; different layered images contain pixels with different grayscale values.

[0119] Determine the number of local minima contained in pixels of two adjacent layered images;

[0120] When the number of local minima is equal to a preset value, the two adjacent layered images are respectively determined as grayscale images.

[0121] Based on the grayscale values ​​of pixels within the target image, intensity layering of the grayscale values ​​is performed to obtain one or more grayscale images.

[0122] First, the target image can be initially divided into multiple layered images based on the grayscale values ​​of the pixels, for example, by randomly dividing it into multiple layered images. The grayscale values ​​of the pixels in different layered images will be different.

[0123] Then, for the layered image, the grayscale values ​​of pixels in adjacent layered images are iterated to determine the number of local minima in the two adjacent layered images. If the number of local minima is within the preset range, it indicates that the grayscale values ​​of the two adjacent layered images are reasonably separated, and these two layered images can be identified as grayscale images.

[0124] The default value can be 3.

[0125] For example, the step of performing intensity layering based on the grayscale values ​​of the pixels of the target image to obtain Pm-1 grayscale images includes: inserting a layered image into two adjacent layered images when the number of local minima is less than the preset value; and / or merging the two adjacent layered images when the number of local minima is greater than the preset value.

[0126] In some embodiments, the method further includes:

[0127] The image to be processed is subjected to skip convolution to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

[0128] In some embodiments, the target image can be the original captured image, or a preprocessed image of the original image that has undergone noise reduction or other preprocessing. In some embodiments, it can also be an image obtained by skip convolution, which reduces the number of pixels but does not reduce the features.

[0129] By using skip convolution, the number of pixels in the image to be processed can be reduced while preserving the features of the target within the image as much as possible.

[0130] Since the size (number of pixels) of the target image is reduced compared to the size (number of pixels) of the image to be processed, the computational load is reduced in subsequent grayscale intensity layering, feature extraction, and target tracking based on multi-scale structural features, thereby improving the target tracking speed.

[0131] In some embodiments, the method further includes:

[0132] The image region of the original image is divided into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0133] The second type of region of the original image is subjected to target enhancement processing to obtain the image to be processed.

[0134] In this embodiment of the disclosure, the image region is divided into two categories: one category has a small pixel difference between the target and the background, and the other category has a large difference between the background and the target.

[0135] This difference can be represented by the difference between pixel values, such as the difference between the average pixel values ​​of the target and the background.

[0136] Target enhancement is performed on the second type of regions where the difference between the target and the background is small. After target enhancement, the difference between the target and the background in the second type of regions is increased. The enhanced second type of regions and the first type of regions are combined to form the image to be processed. In this way, since the second type of regions with small differences between the background and the target are enhanced, and multi-scale structural feature extraction is performed, the phenomenon of losing tracking due to small differences between some targets and their surrounding background can be reduced, thereby improving the target tracking success rate.

[0137] In some embodiments, the step of extracting multi-scale structural features from the grayscale image to obtain multi-scale structural features includes:

[0138] Each of the grayscale images is processed based on grayscale values ​​to obtain one or more of the following: multidimensional grayscale features, multidimensional grayscale variation features, multidimensional effective patch ratio, patch edge variation features, and target scale distribution features.

[0139] like Figure 3 As shown, this disclosure provides a target tracking method, including:

[0140] S210: Divide the target image into a first type of region and a second type of region; wherein the difference between the target and the background in the first type of region is greater than the difference between the target and the background in the second type of region;

[0141] S220: Perform target enhancement processing on the second type of region of the target image to obtain the image to be processed;

[0142] S230: Perform target tracking based on the image to be processed.

[0143] In this embodiment of the disclosure, before processing the image using machine learning models such as neural networks, the image region of the target image is divided into a first type of region and a second type of region. The specific division of the first and second types of regions is determined based on the difference values ​​between the background and the target within the corresponding image regions.

[0144] This difference can be represented by the difference between pixel values, such as the difference between the average pixel values ​​of the target and the background.

[0145] For the second type of region where the difference between the target and the background is small, target augmentation is performed. This makes the target stand out more relative to the background, thereby improving the tracking success rate when using machine learning models for target tracking in the future.

[0146] In some embodiments, performing target enhancement processing on the second type of region of the target image to obtain the image to be processed includes:

[0147] A high-hat transformation is performed on the second type of region of the target image to obtain the image to be processed for target enhancement.

[0148] In some embodiments, S220 may include: performing a high-hat transformation on a second type of region of the target image to obtain the image to be processed for the target enhancement processing.

[0149] By using this heightening effect, the brightness of the target can be increased, thereby enhancing the target.

[0150] For video captured by a camera, existing algorithms for target tracking and detection suffer from several limitations: region-based tracking algorithms often struggle with complex image scenes and fail to detect targets in heavily occluded videos; feature-based tracking methods often fail to distinguish the features of multiple similar targets when there are many targets in the video image, leading to failure; model-based tracking methods require extensive pre-training to build the model, and the original model becomes unusable when the target changes, resulting in narrow applicability; detection-based tracking algorithms require constant updating of their detectors, leading to high resource consumption and low operating efficiency.

[0151] This embodiment first uses deep learning to classify region images and non-region images. Then, the non-region images are preprocessed to make the target more prominent. The images then enter a training subunit where an adaptive skip convolution method is used to obtain a multi-scale, full-feature minimum image. An intensity-layered method is used to extract image features, and then the intensity-layered image features are used to extract multi-scale structural features of the image. Finally, a support vector machine method is used to separate one or more targets.

[0152] The target tracking module includes a tracking unit, a detection unit, and a learning unit. Its task is to detect whether a new target is a target from the previous time step.

[0153] The results storage and display module receives the path sequence returned by the target tracking module and stores it in a cache or a database with a specified storage structure for easy retrieval and display later.

[0154] This technical solution first employs a neural network classification method to quickly determine whether there is a target to be detected; then, it uses adaptive skip convolution to quickly obtain the minimum image with all features, and uses intensity layering and feature extraction to obtain multi-size structural features, providing minimal feature input for machine learning and quickly achieving the target detection task.

[0155] This invention provides a method for rapid feature extraction in target tracking and detection, which mainly comprises: a video receiving and encoding module, a target detection module, a target tracking module, and a result storage and display module, as follows: Figure 4 As shown in the overall solution framework diagram, the video receiving and encoding module receives video streams captured by the camera or data source, and performs reception and encoding / decoding on the captured video stream; the target detection module performs target detection on the images in the video; the target tracking module tracks the detected targets and corrects the detection process for the next time; and the result storage and display module receives the detection and tracking results and stores and displays them.

[0156] refer to Figure 5As shown, the video receiving and encoding module receives and encodes the video stream captured by the camera. It connects to the IPCAM (Internet Protocol Camera) via its PoE (Power over Ethernet) port to pull the RTSP (Real-Time Streaming Protocol) video stream from the IPCAM. After pulling the RTSP video stream, it encodes it to achieve high compression ratio, high image quality, and good network adaptability. After encoding, it decodes the video stream before passing it to the subsequent video parsing and inference module. The block diagram of this module is as follows. Figure 2 The block diagram of the video receiving and encoding module is shown.

[0157] The image classification unit is used to classify video images, primarily dividing them into two categories: region images and non-region images. Each category has two subcategories: single-target images and multi-target images. Region images are defined as images with a large number of occluded areas (above a set threshold), where the target is not prominent and the target blends well with the background (above a set threshold). Non-region images are defined as images with fewer occluded areas (below a set threshold), where the target is prominent and the contrast between the target and the background is high (above a set threshold). Single-target images are defined as images with fewer targets than a set threshold; multi-target images are defined as images with more targets than a set threshold.

[0158] The image preprocessing subunit performs image preprocessing on images that are classified as non-regional. The main image preprocessing steps include: image noise reduction, image compensation, and image enhancement.

[0159] The training subunit generates small-size images with full features from the images preprocessed by the image preprocessing subunit to obtain the training set, and then performs the following steps:

[0160] A small-size image with full features is obtained by adaptive jump convolution (AJN) using the image and window function W(m,n). The AJN calculation method is as follows:

[0161] Calculate the jump coefficient Rw of the row (i，j) :

[0162]

[0163] Where Tr is a set fluctuation threshold, and s (i，j) The row-level fluctuation quantization function for the i-th row and j-th column is calculated as follows:

[0164]

[0165] in

[0166] Calculate the jump coefficient Cw of the column (i，j) :

[0167]

[0168] Where Tc is a set threshold, and s (i，j) The column volatility quantization function for the i-th row and j-th column is calculated as follows:

[0169]

[0170] in

[0171] Calculate multi-scale images for

[0172] in For example, q (0，0) =(Rw (0，0) &Rw (1，0) &Rw (2，0) &...Rw (R，0) )&Cw (0，2) &Cw (0，2) &Cw (0，2) &...Cw (0，C) ), & is the AND operator.

[0173] r = min(r0, r1, ..., r) j ,...,r m ), where r j satisfy

[0174] c = min(c0, c1, ..., c) j c n ), where c i satisfy

[0175] Equilibrium normalized image, i.e.:

[0176]

[0177] Where 0≤i<C, 0≤j<R, C represents the number of columns in the image, R represents the number of rows in the image, and W(m,n) is an m×n window function.

[0178] The image extraction subunit signal is connected to the training subunit, including: an intensity stratification subunit and an extraction subunit; the intensity stratification subunit is used to perform intensity stratification on the image; the extraction subunit is used to extract the color patch map obtained from the intensity stratification. The method for performing intensity stratification on the image executes the following steps:

[0179] Let different layers of the image be represented by a three-dimensional function: (x, y, f(x, y)), where x and y determine the position of the pixel value in the corresponding grayscale image, and f(x, y) is the grayscale value in the corresponding grayscale image. Let the grayscale level of the grayscale image be [0, L]. max -1], L max The grayscale axis is defined as the plane axis formed by the grayscale levels, representing the maximum value of each grayscale level. Assume that P planes perpendicular to the grayscale axis are defined as {l1, l2, l3, ..., l...} p}, 0 < p < L max P planes divide the grayscale axis into P+1 intensity groups {V1, V2, V3, ..., V...}. k , ..., V p+1 If 1 ≤ k ≤ p + 1, then f(x, y) can be expressed as: f(x, y) = c·V k Where c is the intensity V at level k. k The relevant grayscale coefficients.

[0180] Let the number of interval layers be s*P, where s is the interval coefficient, s∈[0,1]. Sort the sequence of local minima within the grayscale axis and the two intensity planes in ascending order of grayscale value. Let the total number of local minima sequences be P. m The number of intensity layers is P l Then we have:

[0181] When P l =P m When the value is -1, it means that the features of the image match the intensity layering, so the image is directly subjected to intensity layering.

[0182] When P l <P m When -1, it indicates that the image features are more refined than the intensity level, thus coarsening the features. In this case, the feature is selected from the minimum point sequence (P...). m -P l +1) Local minima with larger grayscale ranges are used for intensity layering;

[0183] When P l >P m When -1, it indicates that the image's feature refinement is less than the number of intensity layers. In the local minimum sequence, a complementary value is selected between the two extreme values ​​with larger grayscale ranges (P). l -P m +1) point.

[0184] The feature extraction subunit is used to extract multi-scale structural features of the image, including multi-dimensional gray-level variation features, multi-dimensional effective patch ratio, patch edge variation features, and target volume scale distribution features.

[0185] Before feature extraction, each image is thresholded to obtain a binarized image (img). B , image img B Perform normalized multidimensional feature operations to obtain the segmented image X; the specific method is as follows:

[0186] Multidimensional grayscale feature DE:

[0187] Multidimensional grayscale variation feature DEB: In the formula, g(x, y) is img B The gray value of pixel (x, y), N represents the total number of pixels equal to 1 in the segmented image X, i.e., the number of target points, n represents the total number of layers in the image, and f represents the slope function of the fitted line.

[0188] The detection subunit uses a support vector machine to detect the training set. By detecting the structural features of the training set, it separates one or more targets, thus obtaining the final detection result.

[0189] Target tracking mainly comprises a tracking unit, a detection unit, and a learning unit. The main methods for target tracking include:

[0190] Based on the target detected by the target detection section, the tracking unit uses the median optical flow method to predict the target determined by the target region information in the previous frame image, and uses this prediction as the target tracking result.

[0191] Determine whether the number of pixels occupied by the target region in the previous frame image is greater than the pixel threshold in the current frame image;

[0192] If so, the target region in the previous frame and the current frame are reduced according to a certain aspect ratio, and the current frame is traversed globally using a sliding window to obtain the candidate targets determined by the detection unit.

[0193] If not, the current frame image is divided into local regions, and a sliding window is used to traverse the local regions to obtain the candidate targets determined by the detection unit, with a pixel threshold of T. pix .

[0194] The variance classifier, random classifier, and nearest neighbor classifier of the detection unit are used to process at least one candidate target to form a target detection result; the target tracking region is determined based on the target tracking result and the target detection result to finally complete the target tracking of the current frame.

[0195] Initialize the target region selection from the first frame image, including: in the starting frame of the video or image sequence, manually determine a rectangle containing the target and obtain the initial coordinates and width and height information of the rectangle as the initial target region information.

[0196] The sliding window approach includes: setting the scanning strategy of the sliding window in the current frame image; the scanning strategy is: the displacement of the sliding window in the horizontal and vertical directions is the metric (Ration) of the width and height dimensions of the sliding window, respectively; after each scan, the size of the sliding window is expanded or reduced by t times as needed, and the minimum sliding window size is n.

[0197] The current frame image is divided into local regions, including: using the center position of the target in the previous frame as the center of the local traversal region division, and using the image region formed by t times the width and height of the target in the previous frame as the local region division in the current frame.

[0198] The results storage and display module receives the path sequence returned by the target tracking module and stores it in a cache or a database with a specified storage structure for easy retrieval and display later.

[0199] When performing object detection, an image classification technique is first employed, which differs from traditional classification. This classification process aims to categorize images based on their quality, and then preprocesses the images based on the classification results.

[0200] This approach reduces the likelihood of decreased accuracy in subsequent detection and tracking due to low background contrast, significant occlusion, or numerous targets in the original image. This results in more accurate final classification results. Simultaneously, images with high background contrast are left unprocessed for direct target detection, ensuring that detection efficiency is not compromised.

[0201] This paper proposes a target detection scheme based on image intensity stratification and multi-scale features. It proposes a feature classification method based on the minimum value of the intensity plane and establishes an extraction sub-unit for extracting color patches obtained from intensity stratification. This algorithm is completely different from existing algorithms and achieves a significant improvement in the accuracy of the final detection result.

[0202] An adaptive skip convolution method is proposed, which performs adaptive skip convolution on the preprocessed image to obtain a minimum-size feature image containing all features. The resulting smaller image can improve the efficiency of feature extraction.

[0203] The overall system structure: In this embodiment of the invention, video images acquired by a camera are first obtained through a video receiving and encoding module, and then a video parsing and inference module is used to perform target detection and tracking on the images in the video, thereby realizing the automation and intelligence of video processing.

[0204] The target detection process: In this embodiment of the invention, target detection first involves image classification, which differs from traditional classification. This classification process aims to categorize images based on their quality, and then preprocesses the images based on the classification results. This reduces situations where low background contrast, significant occlusion, or numerous targets in the original image can lead to decreased accuracy in subsequent detection and tracking. This results in a more accurate final classification result. Simultaneously, images with high background contrast are not processed; these images are directly used for target detection, ensuring that detection efficiency is not reduced.

[0205] The target detection algorithm used in this embodiment of the invention is based on image intensity stratification and multi-scale features, which is completely different from existing algorithms. Furthermore, this algorithm significantly improves the accuracy of the final detection results.

[0206] like Figure 6 As shown, this disclosure provides a target tracking device, including:

[0207] The layering module 110 is used to perform intensity layering of the gray values ​​based on the gray values ​​of pixels in the target image to obtain one or more grayscale images.

[0208] Extraction module 120 is used to extract multi-scale structural features from the grayscale image to obtain multi-scale structural features;

[0209] The tracking module 130 is used to track the target based on the multi-scale structural features.

[0210] This target tracking device can be applied to various electronic devices, including but not limited to terminals and / or servers.

[0211] In one embodiment, the layering module 110, the extraction module 120, and the tracking module 130 may be program modules; after being executed by the processor, the program modules can realize the functions of any of the aforementioned modules.

[0212] In another embodiment, the layering module 110, the extraction module 120, and the tracking module 130 may be a hardware-software integrated module; the hardware-software integrated module includes, but is not limited to, various programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0213] In some embodiments, the layering module 110, the extraction module 120, and the tracking module 130 may be pure hardware modules; the pure hardware modules include, but are not limited to, application-specific integrated circuits.

[0214] In some embodiments, the tracking module 130 is specifically used to determine the candidate region of the nth target in the (m+1)th frame image based on the m-th frame image; wherein m and n are both positive integers; determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the (m+1)th frame image; and determine the position of the nth target in the (m+1)th frame image based on the similarity.

[0215] In some embodiments, the tracking module 130 is specifically configured to perform at least one of the following:

[0216] Based on the optical flow estimation of the nth target in the mth frame image, the candidate region of the mth target in the (m+1)th frame image is determined;

[0217] The (m+1)th frame image is divided into multiple image regions based on the s-th sliding window, and the candidate region is determined based on the pixel similarity between the image region and the region where the nth target is located in the m-th frame image; wherein, s is a positive integer less than or equal to S; S is a positive integer not less than 2; and the size of different sliding windows is different.

[0218] In some embodiments, the layering module 110 is specifically used to perform intensity layering based on the gray values ​​of the pixels of the target image to obtain Pm-1 grayscale images, wherein Pm is the number of local minima of the gray values ​​of the pixels of the target image, and the gray values ​​of the pixels of different grayscale images are located between different local minima groups, and a local minima group includes two adjacent local minima.

[0219] In some embodiments, the layering module 110 is further configured to divide the target image into multiple layered images according to the grayscale value of the pixels; different layered images contain pixels with different grayscale values; determine the number of local minima contained in the pixels of two adjacent layered images; and when the number of local minima is equal to a preset value, determine the two adjacent layered images as grayscale images respectively.

[0220] In some embodiments, the layering module 110 is specifically used to insert a layered image into two adjacent layered images when the number of local minima is less than the preset value; and / or to merge two adjacent layered images when the number of local minima is greater than the preset value.

[0221] In some embodiments, the apparatus further includes:

[0222] The skip convolution module is used to perform skip convolution processing on the image to be processed to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

[0223] In some embodiments, the apparatus further includes:

[0224] The segmentation module is used to divide the image region of the original image into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0225] An enhancement module is used to perform target enhancement processing on the second type of region of the original image to obtain the image to be processed.

[0226] In some embodiments, the extraction module 120 is specifically used to process each of the grayscale images based on grayscale values ​​to obtain one or more of the following: multidimensional grayscale features, multidimensional grayscale change features, multidimensional effective patch ratio, patch edge change features, and target scale distribution features.

[0227] like Figure 7 As shown, this disclosure provides a target tracking device, including:

[0228] The region module 210 is used to divide the image region of the target image into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region;

[0229] The image to be processed module 220 is used to perform target enhancement processing on the second type of region of the target image to obtain the image to be processed;

[0230] The target module 230 is used for target tracking based on the image to be processed.

[0231] In some embodiments, the region module 210, the image to be processed module 220, and the target module 230 may be program modules; after being executed by the processor, the program modules can realize the functions of the above-mentioned modules.

[0232] In other embodiments, the region module 210, the module to be processed, and the target module 230 may be a hardware-software hybrid module; the hardware-software hybrid module includes, but is not limited to, various programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0233] In some embodiments, the image to be processed module 220 is specifically used to perform a high-hat transformation on the second type of region of the original image to obtain the image to be processed for the target enhancement processing.

[0234] like Figure 8As shown, this disclosure provides an electronic device, the electronic device comprising:

[0235] Memory;

[0236] The processor, connected to the memory, is configured to implement the target tracking method provided in any of the foregoing embodiments by executing computer-executable instructions stored in the memory, such as performing... Figures 1 to 4 Any text processing method shown.

[0237] The electronic device may be a terminal device and / or a server in a service platform.

[0238] like Figure 8 As shown, the electronic device may also include a network interface, which can be used to interact with peer devices via a network.

[0239] This disclosure provides a computer storage medium storing computer-executable instructions; when executed by a processor, these computer-executable instructions can implement the text processing method provided in any of the foregoing embodiments, for example, performing... Figures 1 to 4 Any target tracking method shown.

[0240] The computer's storage medium is a non-instantaneous storage medium.

[0241] The technical solutions described in the embodiments of this disclosure can be combined arbitrarily without conflict.

[0242] In the several embodiments provided in this disclosure, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0243] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0244] In addition, each functional unit in the various embodiments of this disclosure can be integrated into a second processing unit, or each unit can be a separate unit, or N or more units can be integrated into a unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0245] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A target tracking method, characterized in that, include: Based on the grayscale values ​​of pixels within the target image, intensity layering of the grayscale values ​​is performed to obtain one or more grayscale images; Multi-scale structural features are extracted from the grayscale image to obtain multi-scale structural features; Target tracking is performed based on the aforementioned multi-scale structural features; The process of obtaining one or more grayscale images by intensity layering of the grayscale values ​​based on the grayscale values ​​of pixels within the target image includes: Intensity layering is performed based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images; Wherein, Pm is the number of local minima of the grayscale value of the pixels in the target image, and the grayscale values ​​of pixels in different grayscale images are located between different local minima groups. A local minima group includes two adjacent local minima. The local minima are obtained by performing histogram statistics on the grayscale values ​​of all pixels in the target image. The step of target tracking based on the multi-scale structural features includes: For each of the at least two sliding windows, perform the following operations: divide the (m+1)th frame image into multiple image regions based on the sliding window; wherein, the size of the sliding window is determined by scaling the initial sliding window according to a corresponding scaling ratio, and the size of the initial sliding window is determined according to the image size of the nth target in the mth frame image, and the sizes of different sliding windows are different; m and n are both positive integers; Based on the pixel similarity between each of the image regions and the region where the nth target is located in the mth frame image, the candidate region of the nth target in the m+1th frame image is determined. Determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the (m+1)-th frame image; Based on the similarity, the position of the nth target in the (m+1)th frame of the image is determined.

2. The method according to claim 1, characterized in that, The step of performing intensity layering based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images includes: Based on the grayscale values ​​of the pixels, the target image is divided into multiple layered images; different layered images contain pixels with different grayscale values. Determine the number of local minima contained in pixels of two adjacent layered images; When the number of local minima is equal to a preset value, the two adjacent layered images are respectively determined as grayscale images.

3. The method according to claim 2, characterized in that, The step of performing intensity layering based on the grayscale values ​​of the pixels in the target image to obtain Pm-1 grayscale images further includes: When the number of local minima is less than the preset value, a layered image is inserted between two adjacent layered images; And / or, When the number of local minima is greater than the preset value, the two adjacent layered images are merged.

4. The method according to claim 1, characterized in that, The method further includes: The image to be processed is subjected to skip convolution to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

5. The method according to claim 4, characterized in that, The method further includes: The image region of the original image is divided into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region; The second type of region of the original image is subjected to target enhancement processing to obtain the image to be processed.

6. The method according to claim 5, characterized in that, The process of performing target enhancement processing on the second type of region of the original image to obtain the image to be processed includes: A high-hat transformation is performed on the second type of region of the original image to obtain the image to be processed for the target enhancement process.

7. The method according to any one of claims 1 to 6, characterized in that, The step of extracting multi-scale structural features from the grayscale image to obtain multi-scale structural features includes: Each of the grayscale images is processed based on grayscale values ​​to obtain one or more of the following: multidimensional grayscale variation features, multidimensional effective patch ratio, patch edge variation features, and target scale distribution features.

8. A target tracking device, characterized in that, include: A layering module is used to perform intensity layering of gray values ​​based on the gray values ​​of pixels within a target image to obtain one or more grayscale images. The extraction module is used to extract multi-scale structural features from the grayscale image to obtain multi-scale structural features; The tracking module is used to track targets based on the multi-scale structural features; The layering module is specifically used to perform intensity layering based on the grayscale values ​​of the pixels in the target image, resulting in Pm-1 grayscale images; wherein, Pm is the number of local minima of the grayscale values ​​of the pixels in the target image, and the grayscale values ​​of pixels in different grayscale images are located between different local minima groups, and a local minima group includes two adjacent local minima; the local minima are obtained by performing histogram statistics on the grayscale values ​​of all pixels in the target image; The tracking module is specifically configured to perform the following operations for each of the at least two sliding windows: dividing the (m+1)th frame image into multiple image regions based on the sliding window; wherein, the size of the sliding window is determined by scaling the initial sliding window according to a corresponding scaling ratio, and the size of the initial sliding window is determined according to the image size of the nth target in the mth frame image, and the sizes of different sliding windows are different; m and n are both positive integers; Based on the pixel similarity between each of the image regions and the region where the nth target is located in the mth frame image, the candidate region of the nth target in the m+1th frame image is determined. Determine the multi-scale structural features of the nth target in the m-th frame image, and the similarity between the multi-scale structural features extracted from the candidate region of the (m+1)-th frame image; Based on the similarity, the position of the nth target in the (m+1)th frame of the image is determined.

9. The apparatus according to claim 8, characterized in that, The layering module is further configured to divide the target image into multiple layered images based on the grayscale values ​​of the pixels; different layered images contain pixels with different grayscale values; determine the number of local minima contained in the pixels of two adjacent layered images; and when the number of local minima is equal to a preset value, determine the two adjacent layered images as grayscale images respectively.

10. The apparatus according to claim 9, characterized in that, The layering module is specifically used to insert a layered image into two adjacent layered images when the number of local minima is less than the preset value; and / or to merge two adjacent layered images when the number of local minima is greater than the preset value.

11. The apparatus according to claim 8, characterized in that, The device further includes: The skip convolution module is used to perform skip convolution processing on the image to be processed to obtain the target image; wherein the number of pixels in the target image is less than the number of pixels in the image to be processed.

12. The apparatus according to claim 11, characterized in that, The device further includes: The segmentation module is used to divide the image region of the original image into a first type of region and a second type of region; wherein the difference value between the target and the background in the first type of region is greater than the difference value between the target and the background in the second type of region; An enhancement module is used to perform target enhancement processing on the second type of region of the original image to obtain the image to be processed.

13. The apparatus according to claim 12, characterized in that, The enhancement module is specifically used to perform a high-hat transformation on the second type of region of the original image to obtain the image to be processed for the target enhancement processing.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The extraction module is specifically used to process each grayscale image based on grayscale values ​​to obtain one or more of the following: multidimensional grayscale variation features, multidimensional effective patch ratio, patch edge variation features, and target scale distribution features.

15. An electronic device, characterized in that, The electronic device includes: Memory; A processor, connected to the memory, is configured to implement the target tracking method provided by any one of claims 1 to 7 by executing computer-executable instructions stored in the memory.

16. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; when the computer-executable instructions are executed by the processor, they can implement the target tracking method provided in any one of claims 1 to 7.

Images