Target detection and tracking method and device for video image and electronic equipment

Abstract

The invention provides a target detection and tracking method and device for a video image and electronic equipment. Comprising the steps of obtaining a to-be-detected image, cutting the to-be-detected image into a preset number of to-be-detected sub-images, inputting the to-be-detected sub-images into a Detector model, generating a thermodynamic diagram and a rough detection image, cutting each to-be-detected sub-image by adopting an Aug-LSM algorithm, and generating a candidate region corresponding to each to-be-detected sub-image; and inputting each candidate area into a Detector model to generate an accurate detection image corresponding to each candidate area, performing fusion processing on the accurate detection image and the rough detection image to obtain a target detection image, and realizing target tracking by adopting a DeepSORT target tracking algorithm according to the target detection image. And the cut sub-images are processed by adopting an Aug-LSM algorithm, so that the detection precision and robustness of the small object of the image are improved. The problem that a traditional detector is difficult to accurately recognize tiny objects in an image is solved.

Description

Technical Field

[0001] The present application relates to the technical field of target detection and tracking of video images, and in particular to a method for detecting a target in a video image, a device for detecting a target in a video image, a computer-readable storage medium, and an electronic device. Background Art

[0002] Object detection is an important task in the field of computer vision, which aims to identify target objects in images or videos and accurately locate their positions in the images. Current object detection can be divided into two categories, one is two-stage detection and the other is single-stage detection. Two-stage detection method: In the first step, the model generates a series of candidate regions (Region of Interest, RoI) that may contain objects, which are usually generated by specialized network modules. The second step is to perform more detailed analysis in each candidate region for accurate classification and bounding box regression. Single-stage detection method: The model directly predicts the location and category of the object from the input image without generating candidate regions first. As the first step to achieve tracking tasks, the importance of object detection is self-evident. The detection accuracy directly affects the performance of subsequent tracking algorithms.

[0003] Tiny objects occupy fewer pixels in an image, resulting in low resolution and unclear visual features, making it difficult for traditional detectors to accurately identify them. Summary of the invention

[0004] The main purpose of the present application is to provide a method for object detection in video images, an object detection device in video images, a computer-readable storage medium and an electronic device, so as to at least solve the problem that traditional detectors are difficult to accurately identify tiny objects in images.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for target detection and tracking of video images is provided, including: obtaining an image to be detected, and dividing the image to be detected into a preset number of sub-images to be detected based on a center point; inputting the preset number of sub-images to be detected into a Detector model respectively, generating a thermal map and a rough detection image corresponding to each sub-image to be detected, and using the Aug-LSM algorithm to crop each sub-image to be detected according to the thermal map corresponding to each sub-image to be detected, to generate a candidate area corresponding to each sub-image to be detected; inputting each candidate area into the Detector model, generating a precise detection image corresponding to each candidate area, and fusing the precise detection image and the rough detection image to obtain a target detection image, and using the DeepSORT target tracking algorithm to implement target tracking based on the target detection image.

[0006] Optionally, target tracking is achieved using the DeepSORT target tracking algorithm based on the target detection image, including: extracting a first appearance feature vector of the detection target in the current frame target detection image, wherein the appearance feature vector is used to describe the appearance characteristics of the detection target; updating a Kalman filter based on the first appearance feature vector of the detection target, and using the Kalman filter to predict position information of the detection target in the next frame target detection image to generate a prediction result; acquiring the next frame target detection image, extracting a second appearance feature vector of the next frame target detection image, matching the first appearance feature vector with the second appearance feature vector, and matching the prediction result with the position information of the detection target in the next frame target detection image to generate a matching result; and updating the Kalman filter based on the matching result to achieve continuous tracking of the detection target.

[0007] Optionally, updating the Kalman filter according to the matching result includes: when it is determined that the matching result indicates that the first appearance feature vector successfully matches the second appearance feature vector, and the prediction result successfully matches the position information of the detection target in the next frame target detection image, updating the Kalman filter based on the second appearance feature vector and the prediction result.

[0008] Optionally, updating the Kalman filter according to the matching result includes: initializing the Kalman filter based on the second appearance feature vector and the position information of the detection target in the next frame target detection image when it is determined that the matching result indicates that the first appearance feature vector fails to match the second appearance feature vector, and / or the prediction result fails to match the position information of the detection target in the next frame target detection image.

[0009] Optionally, determining that the matching result indicates that the first appearance feature vector and the second appearance feature vector are successfully matched includes: determining a feature similarity between the first appearance feature vector and the second appearance feature vector; and determining that the first appearance feature vector and the second appearance feature vector are successfully matched when it is determined that the feature similarity is greater than a preset similarity.

[0010] Optionally, after generating the candidate regions corresponding to each of the sub-images to be detected, the method further includes: assigning a corresponding confidence score to each candidate grid in the candidate region; and selecting a preset number of the candidate regions according to the confidence scores and inputting them into the Detector model.

[0011] Optionally, extracting a first appearance feature vector of a detection target in a current frame target detection image includes: extracting the first appearance feature vector of the detection target in the current frame target detection image using a deep neural network.

[0012] According to another aspect of the present application, a target detection and tracking device for a video image is provided, comprising: an acquisition unit, used to acquire an image to be detected, and to equally divide and crop the image to be detected into a preset number of sub-images to be detected based on a center point; a processing unit, used to input the preset number of sub-images to be detected into a Detector model respectively, to generate a thermal map and a rough detection image corresponding to each of the sub-images to be detected, and to crop each of the sub-images to be detected using an Aug-LSM algorithm based on the thermal map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected; a tracking unit, used to input each of the candidate regions into the Detector model, to generate a precise detection image corresponding to each of the candidate regions, to fuse the precise detection image and the rough detection image to obtain a target detection image, and to implement target tracking based on the target detection image using a DeepSORT target tracking algorithm.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for object detection and tracking of video images.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the target detection and tracking methods for video images.

[0015] Apply the technical solution of the present application, obtain the image to be detected, and divide the image to be detected into a preset number of sub-images to be detected based on the center point; input the preset number of sub-images to be detected into the Detector model respectively, generate the thermal map and rough detection image corresponding to each sub-image to be detected, and use the Aug-LSM algorithm to crop each sub-image to be detected according to the thermal map corresponding to each sub-image to be detected, and generate the candidate area corresponding to each sub-image to be detected; input each candidate area into the Detector model, generate the precise detection image corresponding to each candidate area, and fuse the precise detection image and the rough detection image to obtain the target detection image, and use the DeepSORT target tracking algorithm to achieve target tracking based on the target detection image. By using the Aug-LSM algorithm to process the cropped sub-images, the detection accuracy is enhanced, the possibility of missed detection and false detection is reduced, the accuracy of the model is improved, and the detection accuracy and robustness of small objects in the image are improved. Solve the problem that traditional detectors are difficult to accurately identify tiny objects in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for object detection and tracking of video images provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process of a method for object detection and tracking of a video image provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a process of a method for detecting and tracking a target in a specific video image provided in an embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of a Detector detection provided according to an embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram of a process for implementing target tracking using the DeepSORT target tracking algorithm provided in an embodiment of the present application is shown;

[0022] Figure 6 A structural block diagram of a device for object detection and tracking of video images provided according to an embodiment of the present application is shown.

[0023] The above drawings include the following reference numerals:

[0024] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] As introduced in the background technology, it is difficult for traditional detectors to accurately identify tiny objects in images. In order to solve the problem that traditional detectors are difficult to accurately identify tiny objects in images, the embodiments of the present application provide a target detection method for video images, a target detection device for video images, a computer-readable storage medium and an electronic device.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for detecting and tracking a target in a video image according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for object detection and tracking of video images in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a method for target detection and tracking of video images running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 FIG. 1 is a flow chart of a method for detecting and tracking a target in a video image according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0034] Step S201, obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point;

[0035] Among them, the preset number can be set to 4;

[0036] Specifically, the input image is precisely divided into four equal parts based on the center point, thereby obtaining four sub-images of equal area. This step ensures the uniformity and symmetry of the image in subsequent processing, laying the foundation for subsequent detection and cropping.

[0037] Step S202, respectively inputting the preset number of sub-images to be detected into the Detector model, generating a heat map and a rough detection image corresponding to each of the sub-images to be detected, and using the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected;

[0038] Specifically, the Augmented-LSM (Aug-LSM) algorithm is an image processing technique based on the least squares method, which is used to enhance the quality and details of an image. The algorithm enhances the image to make it clearer, fuller and more realistic.

[0039] In step S203, each of the candidate regions is input into the Detector model to generate a precise detection image corresponding to each of the candidate regions, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

[0040] Through this embodiment, the image to be detected is obtained, and the image to be detected is equally divided and cropped into a preset number of sub-images to be detected based on the center point; the preset number of sub-images to be detected are respectively input into the Detector model to generate the thermal map and rough detection image corresponding to each sub-image to be detected, and the Aug-LSM algorithm is used to crop each sub-image to be detected according to the thermal map corresponding to each sub-image to be detected, and the candidate area corresponding to each sub-image to be detected is generated; each candidate area is input into the Detector model to generate the precise detection image corresponding to each candidate area, and the precise detection image and the rough detection image are fused to obtain the target detection image, and the DeepSORT target tracking algorithm is used to realize target tracking according to the target detection image. By using the Aug-LSM algorithm to process the cropped sub-image, the detection accuracy is enhanced, the possibility of missed detection and false detection is reduced, the accuracy of the model is improved, and the detection accuracy and robustness of small objects in the image are improved. The problem that traditional detectors are difficult to accurately identify tiny objects in images is solved.

[0041] In the specific implementation process, the DeepSORT target tracking algorithm is used to realize target tracking according to the target detection image, including: extracting the first appearance feature vector of the detection target in the current frame target detection image, wherein the appearance feature vector is used to describe the appearance characteristics of the detection target; updating the Kalman filter according to the first appearance feature vector of the detection target, and using the Kalman filter to predict the position information of the detection target in the next frame target detection image to generate a prediction result; obtaining the next frame target detection image, extracting the second appearance feature vector of the next frame target detection image, matching the first appearance feature vector with the second appearance feature vector, and matching the prediction result with the position information of the detection target in the next frame target detection image to generate a matching result; updating the Kalman filter according to the matching result to realize continuous tracking of the detection target.

[0042] This method, DeepSORT (Deep learning based SORT), is a visual target tracking algorithm based on deep learning. It combines deep learning with the traditional target tracking algorithm SORT (Simple Online and Realtime Tracking). DeepSORT detects the target in each frame image based on the target detector (such as YOLO, Faster R-CNN, etc.), and uses the multi-feature fusion technology to represent and describe the target, and then uses the SORT algorithm to track the target. Based on the SORT algorithm, DeepSORT introduces the Re-IDentification (Re-ID) model to solve the problem of determining the target ID. The Re-ID model determines the unique ID of the target by calculating the similarity of the target in multiple frame images. Kalman filtering plays an important role. The main function of this algorithm is to use the current series of motion variables to predict the motion variables at the next moment, but the first detection result is used to initialize the motion variables of the Kalman filter.

[0043] Specifically, updating the Kalman filter according to the matching result includes: when it is determined that the matching result indicates that the first appearance feature vector successfully matches the second appearance feature vector, and the prediction result successfully matches the position information of the detection target in the next frame target detection image, updating the Kalman filter based on the second appearance feature vector and the prediction result.

[0044] More specifically, updating the Kalman filter according to the matching result includes: initializing the Kalman filter based on the second appearance feature vector and the position information of the detection target in the next frame target detection image when determining that the matching result indicates that the first appearance feature vector fails to match the second appearance feature vector, and / or the prediction result fails to match the position information of the detection target in the next frame target detection image.

[0045] The location information also includes the location coordinates and movement speed information of the target;

[0046] This method matches the predicted position of the Kalman filter with the detected target position and appearance features. During the matching process, the algorithm first calculates the similarity based on the appearance features, and performs supplementary matching in combination with motion information (such as position and speed) when necessary. If the match is successful, the Kalman filter state and appearance features of the target are updated. If not, a new Kalman filter is initialized for the newly detected target. The above steps are repeated to achieve efficient and continuous tracking of the target in the video. The advantage of DeepSORT is that it combines the appearance features extracted by deep learning and the motion prediction ability of the Kalman filter, and can still maintain robustness and high accuracy in the case of occlusion or target overlap.

[0047] Further, determining that the matching result indicates that the first appearance feature vector successfully matches the second appearance feature vector includes: determining a feature similarity between the first appearance feature vector and the second appearance feature vector; and determining that the first appearance feature vector successfully matches the second appearance feature vector when it is determined that the feature similarity is greater than a preset similarity.

[0048] The preset similarity may be set to 80%. When the detected similarity is greater than 80%, the match is considered successful. When the detected similarity is less than or equal to 80%, the match is considered failed.

[0049] Furthermore, after generating the candidate regions corresponding to each of the above-mentioned sub-images to be detected, the above-mentioned method also includes: assigning a corresponding confidence score to each candidate grid in the above-mentioned candidate region; and selecting a preset number of the above-mentioned candidate regions according to the above-mentioned confidence scores and inputting them into the above-mentioned Detector model.

[0050] This method assigns a confidence score to each candidate grid, which can be calculated by the average heat value of each grid and the heat value of the adjacent grids. Regions with high confidence are more likely to contain targets, so regions with high confidence are given priority when generating candidate regions. When the candidate regions are finally selected, the number k is dynamically adjusted according to the confidence distribution of the regions. For example, when the number of high-confidence regions detected is small, the top-K restrictions can be relaxed to ensure that possible targets are not missed; and when the confidence of most regions is high, the number of candidate regions is reduced to increase the detection speed. In addition, an edge extension strategy is added: through a dynamic extension method based on edge pixels, the boundaries of high-confidence regions are further enlarged to ensure the integrity of the included targets. The extension boundary of the region can be set according to the rate of change of the edge confidence of the region.

[0051] Specifically, extracting a first appearance feature vector of a detection target in a current frame target detection image includes: using a deep neural network to extract the first appearance feature vector of the detection target in the current frame target detection image.

[0052] Among them, deep neural network models include: ResNet, VGG, MobileNet, etc.

[0053] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the object detection and tracking method of the video image of the present application will be described in detail below in combination with specific embodiments.

[0054] This embodiment relates to a specific method for detecting and tracking objects in video images. Figure 3 As shown, the following steps are included:

[0055] Step S1, image preprocessing: the input image is precisely divided into four equal parts based on the center point, so as to obtain four sub-images of equal area. This step ensures the uniformity and symmetry of the image in subsequent processing, laying the foundation for subsequent detection and cropping.

[0056] Step S2, Detector detection: Put the four images into the Detector to generate heat maps and regression branches. The schematic diagram of the Detector detection is as follows: Figure 4 shown.

[0057] Step S3, improve the LSM algorithm cropping application in YOLC: On each heatmap generated by S2, Aug-LSM (adaptive local search algorithm) is used to generate crops. The local scale module (LSM) introduces regional confidence weighting and dynamic candidate quantity adjustment to enhance the quality and accuracy of candidate regions, especially in scenes with dense or complex backgrounds.

[0058] Among them, YOLC follows a similar framework to CenterNet, but it differs from CenterNet in that it uses different backbones, detection heads, regression methods, and loss functions. In particular, HRNet is used as a backbone to generate high-resolution heat maps that are better at detecting small objects. In addition, due to the imbalanced distribution of objects in aerial images, a local scale module (LSM) is designed to adaptively search for clustering regions. After detection of the original image and cropping, in dense areas, the precise results are directly replaced with those of the original image.

[0059] YOLC uses higher resolution heatmaps. Specifically, a convolutional layer and two transposed convolutional layers are added to enlarge the heatmap to the same size as the input image. This enables capturing more detailed information about small objects, resulting in more accurate object detection in dense areas. Applying a Gaussian filter before decoding helps reduce false positive predictions in CenterNet. The filter smoothes the heatmap and suppresses multiple peaks around the object. This approach helps improve the localization accuracy of the object and reduces the chance of misclassification.

[0060] Improved LSM——Aug-LSM:

[0061] 1. Weighting of candidate region crop confidence:

[0062] Each candidate grid of each candidate region is assigned a confidence score, which can be calculated by the average heat value of each grid and the heat value of the adjacent grids. Regions with high confidence are more likely to contain targets, so regions with high confidence are selected first when generating candidate regions. Confidence weighting ensures that more important regions are selected first, making the candidate regions higher in quality and reducing false detections of irrelevant regions.

[0063] 2. Dynamically adjust the number of candidate area crops:

[0064] When finally selecting candidate regions, the number k is dynamically adjusted according to the confidence distribution of the regions. For example, when the number of detected high-confidence regions is small, the top-K restrictions can be relaxed to ensure that possible targets are not missed; and when most regions have high confidence, the number of candidate regions is reduced to increase the detection speed.

[0065] 3. Edge expansion strategy:

[0066] Through the dynamic expansion method based on edge pixels, the boundaries of high-confidence regions are further enlarged to ensure the integrity of the target. The expansion boundary of the region can be set according to the rate of change of the edge confidence of the region. The edge expansion strategy ensures that the edges of high-confidence regions are appropriately expanded to capture the entire target object and improve the integrity of the detection.

[0067] Step S4, result fusion: The results of the four-equal-division cropping in S3 are fused to generate the final detection result. This step not only enhances the accuracy of the detection, but also ensures the integrity and reliability of the results.

[0068] Step S5, outputting the final result: outputting the final cropping result after fusion as the image processing solution provided by this embodiment.

[0069] Among them, the loss function of YOLC+ includes the classification loss of the center point, the size regression loss and the offset regression loss.

[0070] YOLC uses the Gaussian Wasserstein distance (GWD) metric to measure the similarity between two bounding boxes. Specifically, given a bounding box B(x, y, h, w) where (x, y) represents the center coordinates, w and h represent the width and height respectively. First, convert it into a 2-D Gaussian distribution. According to GWD, the converted Gaussian distribution is expressed as follows:

[0071] where μ and Σ represent the mean vector and covariance matrix of the Gaussian distribution, respectively.

[0072] μ and Σ are:

[0073] According to the optimal transport theory, the Wasserstein distance between two distributions μ and ν can be calculated as:

[0074]

[0075] Iterate over all R n ×R n A random vector (X,Y) in , such that X ~ μ and Y ~ ν. Therefore, given two 2D Gaussian distributions N(μ1,Σ1) and N(μ2,Σ2), the Wasserstein distance is calculated as:

[0076]

[0077] where ||·|| F represents the Frobenius norm.

[0078]

[0079] Using only the above loss may be sensitive to large errors, therefore, a nonlinear function is performed and the loss is converted to an affinity metric. The final GWD-based loss is expressed as:

[0080] Where f(·) is a nonlinear function that makes the Wasserstein distance smoother and easier to optimize the model. Define f(x) = ln(x+1). T is a hyperparameter that is empirically set to 1.

[0081] Since the GWD-based loss has already taken the center offset into account, the GWD loss is combined with the L1 loss. Specifically, the loss function is: L det =L k +λ gwd L gwd +λ 11 L1;

[0082] Among them, L k is the classification loss of the center point, using the modified focal loss in CenterNet. gwd is a loss based on Gaussian Wasserstein distance, which is used to measure the similarity between the predicted bounding box and the true bounding box. L1 is the L1 loss of size regression, which is used to regress the size of the bounding box. gwd , 11 is a weight parameter used to balance the contribution of different loss terms. gwd is 2, λ 11 is 0.5.

[0083]

[0084] Among them, α and β are two hyperparameters used to balance the difficult and easy samples, and α=2, β=4; Y xyc Indicates the GT value, Represents the predicted value; N represents the number of key points.

[0085] L1 loss, also known as absolute value loss or mean absolute error, is a loss function commonly used for regression problems. It calculates the average of the absolute values ​​of the difference between the predicted value and the true value. In object detection tasks, L1 loss is often used to regress the size of the bounding box, that is, the average of the absolute values ​​of the difference between the width and height of the predicted bounding box and the width and height of the true bounding box. This allows the model to learn a more accurate bounding box size.

[0086] For four equally divided pictures, the loss of the first picture is L det_1 =L k +λ gwd L gwd +λ 11 L1;

[0087] The total loss function is L all =αL det_1 +βL det_2 +μL det_3 +ηL det_4 , where α, β, μ, and η are weight parameters. When there are too many detection targets in one of the images, its weight parameter value is relatively high; otherwise, it is relatively low, and α+β+μ+η=1.

[0088] DeepSORT (Deep learning based SORT) is a visual target tracking algorithm based on deep learning. It combines deep learning with the traditional target tracking algorithm SORT (Simple Online and Realtime Tracking). DeepSORT detects the target in each frame based on the target detector (such as YOLO, Faster R-CNN, etc.), and uses the multi-feature fusion technology to represent and describe the target, and then uses the SORT algorithm to track the target. Based on the SORT algorithm, DeepSORT introduces the Re-IDentification (Re-ID) model to solve the problem of determining the target ID. The Re-ID model determines the unique ID of the target by calculating the similarity of the target in multiple frame images. Kalman filtering plays an important role. The main function of this algorithm is to use the current series of motion variables to predict the motion variables at the next moment, but the first detection result is used to initialize the motion variables of the Kalman filter.

[0089] The flowchart of using DeepSORT target tracking algorithm to achieve target tracking is as follows: Figure 5 As shown in the figure, first, the algorithm detects the target in the input video frame and outputs the bounding box of the target and its category information. Subsequently, a deep neural network is used to extract a high-dimensional appearance feature vector for each detected target to describe the appearance characteristics of the target. Next, a Kalman filter is initialized for each target to model the motion state of the target (such as position, speed, etc.) and predict the target position in subsequent frames. The predicted position of the Kalman filter is matched with the detected target position and appearance features. During the matching process, the algorithm preferentially calculates the similarity based on the appearance features, and performs supplementary matching in combination with motion information (such as position and speed) when necessary. If the match is successful, the Kalman filter state and appearance features of the target are updated; if not, a new Kalman filter is initialized for the newly detected target. The above steps are repeated to achieve efficient and continuous tracking of the target in the video. The advantage of DeepSORT is that it combines the appearance features extracted by deep learning with the motion prediction ability of the Kalman filter, and can still maintain robustness and high accuracy in the case of occlusion or target overlap.

[0090] This embodiment uses the YOLC+ target detection algorithm and Aug-LSM (adaptive local search algorithm) to generate crops, which improves the detection accuracy and robustness of small objects. In complex scenes, objects of the same category may overlap. By utilizing the powerful association ability of the DeepSORT algorithm, combined with target motion information and appearance features, accurate detection and distinction of overlapping objects are achieved.

[0091] The embodiment of the present application also provides a target detection and tracking device for video images. It should be noted that the target detection and tracking device for video images in the embodiment of the present application can be used to execute the target detection and tracking method for video images provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0092] The following is an introduction to the object detection and tracking device for video images provided in an embodiment of the present application.

[0093] Figure 6 Schematic diagram of a device for detecting and tracking a target in a video image according to an embodiment of the present application. Figure 6 As shown, the device comprises:

[0094] An acquisition unit 61 is used to acquire an image to be detected, and to equally divide the image to be detected into a preset number of sub-images to be detected based on a central point.

[0095] The processing unit 62 is used to input the preset number of the sub-images to be detected into the Detector model respectively, generate a heat map and a rough detection image corresponding to each of the sub-images to be detected, and use the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, so as to generate a candidate region corresponding to each of the sub-images to be detected;

[0096] The tracking unit 63 is used to input each of the above-mentioned candidate areas into the above-mentioned Detector model, generate a precise detection image corresponding to each of the above-mentioned candidate areas, fuse the above-mentioned precise detection image with the above-mentioned rough detection image to obtain a target detection image, and use the DeepSORT target tracking algorithm to realize target tracking based on the above-mentioned target detection image.

[0097] In this embodiment, the acquisition unit is used to acquire the image to be detected, and to equally divide and crop the image to be detected into a preset number of sub-images to be detected based on the center point; the processing unit is used to input the preset number of sub-images to be detected into the Detector model, generate the thermal map and rough detection image corresponding to each sub-image to be detected, and use the Aug-LSM algorithm to crop each sub-image to be detected according to the thermal map corresponding to each sub-image to be detected, and generate the candidate area corresponding to each sub-image to be detected; the tracking unit is used to input each candidate area into the Detector model, generate the precise detection image corresponding to each candidate area, and fuse the precise detection image and the rough detection image to obtain the target detection image, and use the DeepSORT target tracking algorithm to realize target tracking according to the target detection image. By using the Aug-LSM algorithm to process the cropped sub-image, the detection accuracy is enhanced, the possibility of missed detection and false detection is reduced, the accuracy of the model is improved, and the detection accuracy and robustness of small objects in the image are improved. The problem that traditional detectors are difficult to accurately identify tiny objects in images is solved.

[0098] As an optional solution, the tracking unit includes a first extraction module, a prediction module, a second extraction module and an update module; the first extraction module is used to extract the first appearance feature vector of the detection target in the current frame target detection image, wherein the appearance feature vector is used to describe the appearance characteristics of the above-mentioned detection target; the prediction module is used to update the Kalman filter according to the above-mentioned first appearance feature vector of the above-mentioned detection target, and use the above-mentioned Kalman filter to predict the position information of the above-mentioned detection target in the next frame target detection image to generate a prediction result; the second extraction module is used to obtain the above-mentioned next frame target detection image, extract the second appearance feature vector of the above-mentioned next frame target detection image, match the above-mentioned first appearance feature vector with the above-mentioned second appearance feature vector, and match the above-mentioned prediction result with the position information of the detection target in the next frame target detection image to generate a matching result; the update module is used to update the above-mentioned Kalman filter according to the above-mentioned matching result to achieve continuous tracking of the above-mentioned detection target.

[0099] An optional solution, the prediction module includes an updating submodule, which is used to update the Kalman filter based on the second appearance feature vector and the prediction result when it is determined that the matching result indicates that the first appearance feature vector successfully matches the second appearance feature vector, and the prediction result successfully matches the position information of the detection target in the next frame target detection image.

[0100] In an optional scheme, the prediction module also includes an initialization module, which is used to initialize the Kalman filter based on the second appearance feature vector and the position information of the detection target in the next frame target detection image when it is determined that the matching result indicates that the first appearance feature vector fails to match the second appearance feature vector, and / or the prediction result fails to match the position information of the detection target in the next frame target detection image.

[0101] An optional scheme, the updating submodule includes a first determining submodule and a second determining submodule, the first determining submodule is used to determine the feature similarity between the first appearance feature vector and the second appearance feature vector; the second determining submodule is used to determine that the first appearance feature vector and the second appearance feature vector are successfully matched when it is determined that the feature similarity is greater than a preset similarity.

[0102] An optional solution, the device also includes an assignment unit and a selection unit; the assignment unit is used to assign a corresponding confidence score to each candidate grid in the above candidate area after generating the candidate area corresponding to each of the above sub-images to be detected; the selection unit is used to select a preset number of the above candidate areas according to the above confidence scores and input them into the above Detector model.

[0103] In an optional solution, the tracking unit includes a third extraction module, which is used to extract the first appearance feature vector of the detection target in the current frame target detection image by using a deep neural network.

[0104] The above-mentioned object detection and tracking device for video images includes a processor and a memory. The above-mentioned acquisition unit, processing unit, tracking unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0105] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem of traditional detectors being difficult to accurately identify tiny objects in images can be solved by adjusting kernel parameters.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the target detection and tracking method for video images.

[0108] Specifically, the object detection and tracking method of the video image includes:

[0109] Step S201, obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point;

[0110] Step S202, respectively inputting the preset number of sub-images to be detected into the Detector model, generating a heat map and a rough detection image corresponding to each of the sub-images to be detected, and using the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected;

[0111] In step S203, each of the candidate regions is input into the Detector model to generate a precise detection image corresponding to each of the candidate regions, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

[0112] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the program executes the object detection and tracking method of the video image when running.

[0113] Specifically, the object detection and tracking method of the video image includes:

[0114] Step S201, obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point;

[0115] Step S202, respectively inputting the preset number of sub-images to be detected into the Detector model, generating a heat map and a rough detection image corresponding to each of the sub-images to be detected, and using the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected;

[0116] In step S203, each of the candidate regions is input into the Detector model to generate a precise detection image corresponding to each of the candidate regions, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

[0117] An embodiment of the present invention provides an electronic device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0118] Step S201, obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point;

[0119] Step S202, respectively inputting the preset number of sub-images to be detected into the Detector model, generating a heat map and a rough detection image corresponding to each of the sub-images to be detected, and using the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected;

[0120] In step S203, each of the candidate regions is input into the Detector model to generate a precise detection image corresponding to each of the candidate regions, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

[0121] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0122] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0123] Step S201, obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point;

[0124] Step S202, respectively inputting the preset number of sub-images to be detected into the Detector model, generating a heat map and a rough detection image corresponding to each of the sub-images to be detected, and using the Aug-LSM algorithm to crop each of the sub-images to be detected according to the heat map corresponding to each of the sub-images to be detected, to generate a candidate region corresponding to each of the sub-images to be detected;

[0125] In step S203, each of the candidate regions is input into the Detector model to generate a precise detection image corresponding to each of the candidate regions, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

[0126] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0132] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0135] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0136] 1) A method for target detection and tracking of a video image of the present application includes: obtaining an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point; inputting the preset number of sub-images to be detected into the Detector model, generating a thermal map and a rough detection image corresponding to each sub-image to be detected, and using the Aug-LSM algorithm to cut and process each sub-image to be detected according to the thermal map corresponding to each sub-image to be detected, and generating a candidate area corresponding to each sub-image to be detected; inputting each candidate area into the Detector model, generating a precise detection image corresponding to each candidate area, and fusing the precise detection image and the rough detection image to obtain a target detection image, and using the DeepSORT target tracking algorithm to achieve target tracking based on the target detection image. By using the Aug-LSM algorithm to process the cropped sub-images, the detection accuracy is enhanced, the possibility of missed detection and false detection is reduced, the accuracy of the model is improved, and the detection accuracy and robustness of small objects in the image are improved. The problem that traditional detectors are difficult to accurately identify tiny objects in images is solved.

[0137] 2) A target detection and tracking device for a video image of the present application comprises: an acquisition unit for acquiring an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on the center point; a processing unit for inputting the preset number of sub-images to be detected into the Detector model, generating a thermal map and a rough detection image corresponding to each sub-image to be detected, and cutting and processing each sub-image to be detected using the Aug-LSM algorithm according to the thermal map corresponding to each sub-image to be detected, and generating a candidate area corresponding to each sub-image to be detected; a tracking unit for inputting each candidate area into the Detector model, generating a precise detection image corresponding to each candidate area, and fusing the precise detection image and the rough detection image to obtain a target detection image, and using the DeepSORT target tracking algorithm to achieve target tracking based on the target detection image. By using the Aug-LSM algorithm to process the cropped sub-images, the detection accuracy is enhanced, the possibility of missed detection and false detection is reduced, the accuracy of the model is improved, and the detection accuracy and robustness of small objects in the image are improved. The problem that traditional detectors are difficult to accurately identify tiny objects in images is solved.

[0138] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for object detection and tracking in video images, characterized in that: include: Acquire an image to be detected, and equally cut the image to be detected into a preset number of sub-images to be detected based on a center point; Input the preset number of sub-images to be detected into the Detector model respectively, generate a heat map and a rough detection image corresponding to each sub-image to be detected, and use the Aug-LSM algorithm to crop each sub-image to be detected according to the heat map corresponding to each sub-image to be detected, and generate a candidate area corresponding to each sub-image to be detected; Each of the candidate regions is input into the Detector model, a precise detection image corresponding to each of the candidate regions is generated, and the precise detection image and the rough detection image are fused to obtain a target detection image, and the target is tracked using the DeepSORT target tracking algorithm based on the target detection image.

2. The method according to claim 1, characterized in that According to the target detection image, DeepSORT target tracking algorithm is used to realize target tracking, including: Extracting a first appearance feature vector of a detection target in a current frame target detection image, wherein the appearance feature vector is used to describe the appearance characteristics of the detection target; updating a Kalman filter according to the first appearance feature vector of the detected target, using the Kalman filter to predict position information of the detected target in a next frame of target detection image, and generating a prediction result; Acquire the next frame target detection image, extract the second appearance feature vector of the next frame target detection image, match the first appearance feature vector with the second appearance feature vector, and match the prediction result with the position information of the detection target in the next frame target detection image to generate a matching result; The Kalman filter is updated according to the matching result to achieve continuous tracking of the detection target.

3. The method according to claim 2, characterized in that Updating the Kalman filter according to the matching result includes: When it is determined that the matching result indicates that the first appearance feature vector successfully matches the second appearance feature vector, and the prediction result successfully matches the position information of the detection target in the next frame target detection image, the Kalman filter is updated based on the second appearance feature vector and the prediction result.

4. The method according to claim 2, characterized in that: Updating the Kalman filter according to the matching result includes: When it is determined that the matching result indicates that the first appearance feature vector fails to match the second appearance feature vector, and / or the prediction result fails to match the position information of the detection target in the next frame target detection image, the Kalman filter is initialized based on the second appearance feature vector and the position information of the detection target in the next frame target detection image.

5. The method according to claim 3, characterized in that: Determining that the matching result indicates that the first appearance feature vector and the second appearance feature vector are successfully matched includes: determining a feature similarity between the first appearance feature vector and the second appearance feature vector; When it is determined that the feature similarity is greater than a preset similarity, it is determined that the first appearance feature vector matches the second appearance feature vector successfully.

6. The method according to claim 1, characterized in that After generating the candidate regions corresponding to the sub-images to be detected, the method further includes: Assigning a corresponding confidence score to each candidate grid in the candidate area; A preset number of the candidate regions are selected according to the confidence scores and input into the Detector model.

7. The method according to claim 2, characterized in that Extracting the first appearance feature vector of the detection target in the current frame target detection image includes: The first appearance feature vector of the detection target in the current frame target detection image is extracted using a deep neural network.

8. A device for detecting and tracking a target in a video image, characterized in that: include: An acquisition unit, used for acquiring an image to be detected, and equally cutting the image to be detected into a preset number of sub-images to be detected based on a center point; A processing unit, used to input the preset number of sub-images to be detected into the Detector model respectively, generate a heat map and a rough detection image corresponding to each sub-image to be detected, and use the Aug-LSM algorithm to crop each sub-image to be detected according to the heat map corresponding to each sub-image to be detected, and generate a candidate area corresponding to each sub-image to be detected; A tracking unit is used to input each of the candidate areas into the Detector model, generate a precise detection image corresponding to each of the candidate areas, fuse the precise detection image with the rough detection image to obtain a target detection image, and implement target tracking using the DeepSORT target tracking algorithm based on the target detection image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the object detection and tracking method for video images according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the target detection and tracking method of the video image described in any one of claims 1 to 7.

Images