A dynamic target analysis method, device, equipment and storage medium

Abstract

The present invention discloses a dynamic target analysis method, device, equipment and storage medium. The method comprises: performing target detection on the video data to be detected, obtaining the target detection frame of each frame image; based on the obtained detection frame, using a preset target tracking algorithm, determining one or more detection frame sets; wherein each detection frame set contains detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets; determining the image quality for the images contained in the detection frames in each detection frame set; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; screening out the detection frame set whose image quality is greater than the preset quality; based on the images contained in the detection frames in each screened detection frame set, determining the category of the target corresponding to the detection frame set.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a dynamic target analysis method, device, equipment and storage medium. Background Art

[0002] Object detection is an important image analysis method that can analyze the location of the target and other related information from the image. For example, the object detection box that represents the target location, the classification of the target, etc.

[0003] In some business needs, it may be necessary to analyze the dynamic situation of the target. For example, when performing target detection on goods, it may be necessary to detect the picking of goods to facilitate the determination of the popularity of the goods. Specifically, it can be to detect the movement of goods and the type of goods.

[0004] However, specific analysis of the target's dynamic situation often requires target detection and other processing on video data, which requires large amounts of calculation and has low detection efficiency. Summary of the invention

[0005] The present invention provides a dynamic target analysis method, device, equipment and storage medium to solve the deficiencies in the related art.

[0006] According to a first aspect of an embodiment of the present invention, a dynamic target analysis method is provided, comprising:

[0007] Perform target detection on the video data to be detected and obtain the target detection frame of each frame image;

[0008] Based on the acquired detection frames, one or more detection frame sets are determined using a preset target tracking algorithm; wherein each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets;

[0009] For the image contained in the detection frame in each detection frame set, determine the image quality; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; and filter out the detection frame set whose image quality is greater than the preset quality;

[0010] Based on the images contained in the detection frames in each of the filtered detection frame sets, the category of the target corresponding to the detection frame set is determined.

[0011] Optionally, determining the category of the target corresponding to the detection frame set based on the image contained in the detection frame in each of the filtered detection frame sets includes:

[0012] Based on the image contained in the detection frame in each filtered detection frame set, the features of the target are extracted, and the features of the target are compared with the features in the preset category feature set. The category corresponding to the features whose similarity is greater than the preset similarity threshold is determined as the category of the target corresponding to the detection frame set.

[0013] Optionally, the performing target detection on the video data to be detected includes:

[0014] Detecting a first category of an object in the video data to be detected;

[0015] The preset classification feature set includes: features of a second category; the second category is a sub-category of the first category;

[0016] The method further comprises: presetting a feature extraction model corresponding to each first category;

[0017] The extracting the features of the target based on the image contained in the detection frame in each of the filtered detection frame sets includes:

[0018] For each filtered detection frame set, determine the first category of the target corresponding to the detection frame set;

[0019] The determined feature extraction model corresponding to the first category is used to extract target features for the image contained in the detection frame in each of the screened detection frame sets.

[0020] Optionally, the performing target detection on the video data to be detected includes:

[0021] Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine a third category of the target; the preset target detection model includes a detection box prediction branch corresponding to each third category;

[0022] When the third category of any target is determined, the detection box output by the detection box prediction branch corresponding to the third category is determined as the detection box of the target.

[0023] Optionally, the performing target detection on the video data to be detected includes:

[0024] Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine the detection frame position, size and rotation angle of the target;

[0025] The preset target detection model includes a regression prediction branch and a classification prediction branch of the detection frame rotation angle;

[0026] For any target, the outputs of the regression prediction branch and the classification prediction branch in the preset target detection model are integrated to determine the comprehensive result of the rotation angle.

[0027] Optionally, the determining one or more detection frame sets based on the acquired detection frame using a preset target tracking algorithm includes:

[0028] The next frame image of the first frame image in the video data to be detected is determined as the current image, and the following steps are performed in a loop:

[0029] Based on the first real detection frame obtained by target detection in the previous frame image, the detection frame in the current image is predicted using preset parameters to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames;

[0030] For each second real detection frame obtained by target detection in the current image, pair it with the obtained predicted detection frame. When it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add it to the detection frame set corresponding to the target;

[0031] Based on the pairing result, updating the preset parameters;

[0032] If the next frame does not exist in the current image, the loop ends;

[0033] When the current image has a next frame image, the next frame image is determined as the current image.

[0034] Optionally, the preset parameters include parameters of Kalman filtering;

[0035] The predicting the detection frame in the current image by using preset parameters includes: predicting the detection frame in the current image by using Kalman filtering;

[0036] The step of pairing each second real detection frame obtained by object detection in the current image with the obtained predicted detection frame includes:

[0037] For each image contained in the second real detection frame obtained by target detection in the current image, and the image contained in the obtained predicted detection frame, a preset detection frame image feature is calculated; the calculation complexity of the preset detection frame image feature is less than a preset complexity threshold;

[0038] Using the Hungarian algorithm, the second real detection frame is paired with the predicted detection frame according to the similarity of the preset detection frame image features and the overlap of the detection frames.

[0039] Optionally, determining the image quality for the image contained in the detection frame in each detection frame set includes:

[0040] Inputting the image contained in the detection frame in each detection frame set into a preset quality analysis model; the preset quality analysis model is used to determine the image quality; the preset quality analysis model includes a regression prediction branch and a classification prediction branch of the image quality;

[0041] For any set of detection frames, the image quality is determined by combining the outputs of the regression prediction branch and the classification prediction branch in the preset quality analysis model.

[0042] Optionally, among the first neural network model for performing target detection on the video data to be detected, the second neural network model for determining the image quality, and the third neural network model for determining the category of the target corresponding to the detection frame set, at least one neural network model is obtained by the following quantization method:

[0043] The initial neural network model trained with the first parameter accuracy is quantized to the second parameter accuracy to obtain an intermediate neural network model; the second parameter accuracy is lower than the first parameter accuracy;

[0044] The following steps are executed repeatedly until the accuracy and computing speed of the current intermediate neural network model meet the preset quantization requirements:

[0045] Determine the accuracy and computational speed of the current intermediate neural network model;

[0046] Determining the output error of each layer between the initial neural network model and the current intermediate neural network model;

[0047] The layer with the largest output error is selected to improve parameter accuracy, and a new intermediate neural network model is obtained as the current intermediate neural network model.

[0048] According to a second aspect of an embodiment of the present invention, there is provided a dynamic target analysis device, comprising:

[0049] The target detection unit is used to perform target detection on the video data to be detected and obtain the target detection frame of each frame image;

[0050] A target tracking unit, configured to determine one or more detection frame sets based on the acquired detection frame and using a preset target tracking algorithm; wherein each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets;

[0051] A screening unit is used to determine the image quality of the image contained in the detection frame in each detection frame set; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; and screen out the detection frame set whose image quality is greater than a preset quality;

[0052] The classification unit is used to determine the category of the target corresponding to each detection frame set based on the image contained in the detection frame in each filtered detection frame set.

[0053] Optionally, the classification unit is used to:

[0054] Based on the image contained in the detection frame in each filtered detection frame set, the features of the target are extracted, and the features of the target are compared with the features in the preset category feature set. The category corresponding to the features whose similarity is greater than the preset similarity threshold is determined as the category of the target corresponding to the detection frame set.

[0055] Optionally, the target detection unit is used to:

[0056] Detecting a first category of an object in the video data to be detected;

[0057] The preset classification feature set includes: features of a second category; the second category is a sub-category of the first category;

[0058] The device is pre-set with feature extraction models corresponding to each first category;

[0059] The classification unit is used for:

[0060] For each filtered detection frame set, determine the first category of the target corresponding to the detection frame set;

[0061] The determined feature extraction model corresponding to the first category is used to extract target features for the image contained in the detection frame in each of the screened detection frame sets.

[0062] Optionally, the target detection unit is used to:

[0063] Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine a third category of the target; the preset target detection model includes a detection box prediction branch corresponding to each third category;

[0064] When the third category of any target is determined, the detection box output by the detection box prediction branch corresponding to the third category is determined as the detection box of the target.

[0065] Optionally, the target detection unit is used to:

[0066] Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine the detection frame position, size and rotation angle of the target;

[0067] The preset target detection model includes a regression prediction branch and a classification prediction branch of the detection frame rotation angle;

[0068] For any target, the outputs of the regression prediction branch and the classification prediction branch in the preset target detection model are integrated to determine the comprehensive result of the rotation angle.

[0069] Optionally, the target tracking unit is used to:

[0070] The next frame image of the first frame image in the video data to be detected is determined as the current image, and the following steps are performed in a loop:

[0071] Based on the first real detection frame obtained by target detection in the previous frame image, the detection frame in the current image is predicted using preset parameters to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames;

[0072] For each second real detection frame obtained by target detection in the current image, pair it with the obtained predicted detection frame. When it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add it to the detection frame set corresponding to the target;

[0073] Based on the pairing result, updating the preset parameters;

[0074] If the next frame does not exist in the current image, the loop ends;

[0075] When the current image has a next frame image, the next frame image is determined as the current image.

[0076] Optionally, the preset parameters include parameters of Kalman filtering;

[0077] The target tracking unit is used to: predict the detection frame in the current image using Kalman filtering;

[0078] For each image contained in the second real detection frame obtained by target detection in the current image, and the image contained in the obtained predicted detection frame, a preset detection frame image feature is calculated; the calculation complexity of the preset detection frame image feature is less than a preset complexity threshold;

[0079] Using the Hungarian algorithm, the second real detection frame is paired with the predicted detection frame according to the similarity of the preset detection frame image features and the overlap of the detection frames.

[0080] Optionally, the screening unit is used to:

[0081] Inputting the image contained in the detection frame in each detection frame set into a preset quality analysis model; the preset quality analysis model is used to determine the image quality; the preset quality analysis model includes a regression prediction branch and a classification prediction branch of the image quality;

[0082] For any set of detection frames, the image quality is determined by combining the outputs of the regression prediction branch and the classification prediction branch in the preset quality analysis model.

[0083] Optionally, among the first neural network model for performing target detection on the video data to be detected, the second neural network model for determining the image quality, and the third neural network model for determining the category of the target corresponding to the detection frame set, at least one neural network model is obtained by the following quantization method:

[0084] The initial neural network model trained with the first parameter accuracy is quantized to the second parameter accuracy to obtain an intermediate neural network model; the second parameter accuracy is lower than the first parameter accuracy;

[0085] The following steps are executed repeatedly until the accuracy and computing speed of the current intermediate neural network model meet the preset quantization requirements:

[0086] Determine the accuracy and computational speed of the current intermediate neural network model;

[0087] Determining the output error of each layer between the initial neural network model and the current intermediate neural network model;

[0088] The layer with the largest output error is selected to improve parameter accuracy, and a new intermediate neural network model is obtained as the current intermediate neural network model.

[0089] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0090] at least one processor; and,

[0091] a memory communicatively connected to the at least one processor; wherein,

[0092] The memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned dynamic target analysis method.

[0093] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned dynamic target analysis method when executed by a processor.

[0094] According to the above embodiments, by determining the image quality of the images contained in the detection frames in each detection frame set, targets with higher image quality are screened out and the target categories are determined. For targets with lower image quality, the step of determining the target category is not performed, thereby saving calculation amount and improving detection efficiency.

[0095] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0097] Figure 1 is a flow chart of a dynamic target analysis method according to an embodiment of the present invention;

[0098] Figure 2 is a flow chart of a target tracking method according to an embodiment of the present invention;

[0099] Figure 3 is a flow chart of a feature extraction method according to an embodiment of the present invention;

[0100] Figure 4 is a flow chart of a model quantization method according to an embodiment of the present invention;

[0101] Figure 5 is a structural schematic diagram of a dynamic target analysis device according to an embodiment of the present invention;

[0102] Figure 6 The figure is a schematic diagram of the hardware structure of a computer device configured with the method of the embodiment of the present invention according to the embodiment of the present invention. DETAILED DESCRIPTION

[0103] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0104] Object detection is an important image analysis method that can analyze the location of the target and other related information from the image. For example, the object detection box that represents the target location, the classification of the target, etc.

[0105] In some business needs, it may be necessary to analyze the dynamic situation of the target. For example, when performing target detection on goods, it may be necessary to detect the picking of goods to facilitate the determination of the popularity of the goods. Specifically, it can be to detect the movement of goods and the type of goods.

[0106] However, specific analysis of the target's dynamic situation often requires target detection and other processing on video data, which requires large amounts of calculation and has low detection efficiency.

[0107] The embodiment of the present invention discloses a dynamic target analysis method. In this method, target detection can be performed for each frame of video data, and then a target tracking algorithm can be used to determine the detection frames belonging to the same target in the video data. Since the detection frame can represent the position information of the target, multiple detection frames belonging to the same target can be used to determine the movement of the target.

[0108] Furthermore, the category of the target may be determined through the images contained in the multiple detection frames that belong to the same target, thereby improving the accuracy of target classification.

[0109] Among them, the amount of calculation consumed in specifically determining the target category is usually large. Therefore, in this method, some targets can be screened out and the subsequent steps of determining the target category are not performed, which saves calculation amount and improves detection efficiency.

[0110] Specifically, if the target is occluded in the image contained in the detection frame, or the target becomes blurred during movement, or the target only corresponds to a few detection frames, the target category determined based on the image contained in the detection frame of the same target has a low accuracy.

[0111] Therefore, the image quality can be determined for the images contained in the detection box belonging to the same target, and then the targets with low image quality can be screened out. Instead of performing the subsequent step of determining the target category, the targets with high image quality are retained and the subsequent step of determining the target category is performed, thereby saving calculation columns and improving detection efficiency. Among them, the image quality can be positively correlated with the classification accuracy of the target. Screening out the targets with low classification accuracy has almost no effect on the overall target classification accuracy.

[0112] Therefore, this method can determine the category of the target by screening out the targets with higher image quality, while not performing the step of determining the target category for the targets with lower image quality. This can save computational effort and improve detection efficiency without affecting the overall target classification accuracy.

[0113] like Figure 1 As shown, Figure 1 It is a flowchart of a dynamic target analysis method according to an embodiment of the present invention.

[0114] The following steps may be included.

[0115] S101: Perform target detection on the video data to be detected, and obtain the target detection frame of each frame image.

[0116] S102: Based on the acquired detection frame, one or more detection frame sets are determined using a preset target tracking algorithm.

[0117] Each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets.

[0118] S103: determining the image quality of the image contained in the detection frame in each detection frame set; and screening out the detection frame sets whose image quality is greater than a preset quality.

[0119] Among them, the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set.

[0120] S104: Determine the category of the target corresponding to each detection frame set based on the image contained in the detection frame in each filtered detection frame set.

[0121] This method process can screen out targets with higher image quality and determine the category of the target by determining the image quality of the images contained in the detection frames in each detection frame set. For targets with lower image quality, the step of determining the target category is not performed, thereby saving calculation amount and improving detection efficiency.

[0122] Since image quality is positively correlated with the classification accuracy of the target, there is a greater possibility of classification errors for the targets with low image quality that are screened out. Therefore, uncertainty about the target category has almost no impact on the overall target classification accuracy.

[0123] The method flow does not limit the operating subject and can be applied to a server or a terminal.

[0124] In an optional embodiment, dynamic target analysis can be achieved through edge computing, which can be specifically applied to edge terminals. However, edge terminals usually have less computing resources. Therefore, this method embodiment can be used to save computing power and improve the efficiency of target detection.

[0125] For ease of understanding, a specific application example is given below.

[0126] The above method flow can be applied to a smart refrigerator, the detected target can be a commodity in the smart refrigerator, and the video data to be detected can be a surveillance video shot inside the smart refrigerator. The smart refrigerator can detect multiple detection frames belonging to the same moving commodity and the category of the commodity that is moving through the above method flow.

[0127] It should be noted that since there are usually many static commodities placed inside smart refrigerators, dynamic target analysis for these static commodities consumes more computing resources. Therefore, the video data to be detected can be the surveillance video taken by the smart refrigerator at the refrigerator delivery port. Since the goods need to pass through the refrigerator delivery port when they are taken out, target detection can be performed more efficiently based on this surveillance video.

[0128] Based on the detection results, the movement trajectory of the product can be determined through multiple detection frames belonging to the same product, and then it can be determined whether the product is taken by the customer.

[0129] By comprehensively analyzing the situation of goods taken and the categories of goods taken, we can analyze the popularity of goods in each category.

[0130] Furthermore, it can be combined with static target detection, specifically, images of the goods inside the smart refrigerator can be periodically captured to determine the changes in the quantity of the goods inside the smart refrigerator, so as to more accurately analyze the popularity of the goods.

[0131] It is also possible to combine customer characteristics, for example, analyze the age and gender of the customer who takes the goods, so as to determine the categories of goods that different types of customers are interested in, and then recommend goods to customers. Specifically, it can be based on different types of customers, and display the types of goods and information recommended to customers on a nearby display screen, such as product prices, discount information, and preferential information.

[0132] The following is a detailed explanation of each step.

[0133] S101: Perform target detection on the video data to be detected, and obtain the target detection frame of each frame image.

[0134] The process of this method does not limit the source of the video data to be detected, and optionally, it can be a surveillance video of the delivery port of the smart refrigerator.

[0135] The process of this method does not limit the specific method of target detection. Optionally, a preset target detection model can be used to perform target detection.

[0136] The process of this method does not limit the specific structure of the target detection model. Optionally, the target detection model can be a model with a small amount of calculation, thereby saving the amount of calculation. The computational complexity of the target detection model can be less than the preset threshold. Specifically, the YOLOv5 target detection model can be used for target detection. Since the YOLOv5 target detection model has a small amount of calculation and occupies less storage resources, it is more suitable for edge computing, can also save the amount of calculation, and improve the efficiency of target detection.

[0137] Of course, other target detection models, for example, other target detection models in the YOLO series, can also be used for target detection in the process of this method.

[0138] In an optional embodiment, since target detection may be performed in S101, the category of the target may also be determined in the target detection. For example, the output target detection frame corresponds to the category of the target.

[0139] Optionally, in order to distinguish the category of the target from that determined in the subsequent step S104, the general category of the target may be determined first during the target detection in S101, and then the subdivided category may be further determined in S104.

[0140] For example, in the scenario of a smart refrigerator, the detected target is specifically a commodity, and commodities can be roughly classified into boxed, bagged, and bottled according to the packaging method. Further, the subcategories may include: boxed beverages, snacks, and toys; bagged beverages, snacks, and toys; and bottled beverages and snacks.

[0141] Therefore, optionally, the video data to be detected can be input into a preset target detection model, and the preset target detection model can be used to determine the third category of the target. This embodiment does not limit the third category, and specifically can be a large category containing subcategories, such as boxed, bagged, and bottled.

[0142] As for the third category of targets, the corresponding detection frame prediction methods may be different due to different third category targets. For example, the prediction methods of the detection frames may need to be differentiated for boxed products with regular shapes and bagged products with irregular shapes.

[0143] Therefore, optionally, in order to improve the accuracy of the detection frame, a detection frame prediction branch corresponding to each third category can be constructed for each third category of the target, so that the detection frame prediction methods of different third categories can be distinguished, greatly improving the accuracy of the target detection frame of each third category.

[0144] Correspondingly, optionally, when specifically training the preset detection model, image samples with detection frame labels and third category labels may be used for training.

[0145] Optionally, performing target detection on the video data to be detected may include: inputting the video data to be detected into a preset target detection model, the preset target detection model may be used to determine the third category of the target; the preset target detection model may include a detection box prediction branch corresponding to each third category. When the third category of any target is determined, the detection box output by the detection box prediction branch corresponding to the third category may be determined as the detection box of the target.

[0146] This embodiment can improve the prediction accuracy of the detection frame by setting independent detection frame prediction branches of different third categories.

[0147] Regarding the detection frame, the method flow does not limit the specific format.

[0148] In an optional embodiment, a rotating detection frame may be used. For example, in a smart refrigerator scenario, the product may rotate as the customer takes it, presenting different sides in the video data. Therefore, a rotating detection frame may be used to improve the accuracy of the detection frame.

[0149] Therefore, optionally, the preset target detection model can be used to determine the position, size and rotation angle of the detection frame of the target. The position, size and rotation angle of the detection frame can be combined to determine the rotation detection frame.

[0150] The rotation angle of the detection frame may specifically be the rotation angle of the long side of the detection frame or the rotation angle of the vertical side, which is not limited in this embodiment.

[0151] Further, optionally, regression prediction may be performed on the rotation angle, or classification prediction may be performed on the rotation angle, specifically, the rotation angle may be divided into 180 categories from 0 to 179 degrees.

[0152] Among them, the regression prediction has higher accuracy, while the classification prediction has better stability and smaller deviation.

[0153] Therefore, optionally, regression prediction and classification prediction for the rotation angle may be combined to obtain a predicted rotation angle, thereby improving the precision, accuracy and stability of the rotation angle.

[0154] Optionally, performing target detection on the video data to be detected may include: inputting the video data to be detected into a preset target detection model, which may be used to determine the detection frame position, size and rotation angle of the target; the preset target detection model may include a regression prediction branch and a classification prediction branch for the rotation angle of the detection frame; for any target, the outputs of the regression prediction branch and the classification prediction branch in the preset target detection model may be combined to determine the comprehensive result of the rotation angle.

[0155] This embodiment can improve the precision, accuracy and stability of the rotation angle by means of comprehensive regression prediction and classification prediction.

[0156] Among them, the comprehensive result of the rotation angle can be determined as the rotation angle of the final output detection frame.

[0157] Therefore, optionally, the detection frame may be output based on the comprehensive results of the detection frame position, size and rotation angle output by a preset target detection model.

[0158] The method flow does not limit the training method of the preset target detection model. Optionally, image samples marked with detection frame labels can be used for training, image samples marked with rotated detection frame labels can be used for training, and image samples marked with detection frame labels and target category labels can be used for training.

[0159] S102: Based on the acquired detection frame, one or more detection frame sets are determined using a preset target tracking algorithm.

[0160] Each detection frame set may include detection frames belonging to the same target between different images, and different detection frame sets may correspond to different targets.

[0161] Optionally, each of the obtained detection frames can be collected into a detection frame set. Since a single detection frame usually corresponds to only one target, there is usually no situation where the same detection frame is included in different detection frame sets.

[0162] The process of this method is not limited to a specific preset target tracking algorithm, as long as the determined detection frame set meets the above requirements, that is, each detection frame set can contain detection frames belonging to the same target between different images, and different detection frame sets can correspond to different targets.

[0163] In an optional embodiment, for the video data to be detected, the detection frames of two adjacent frames of images may be compared frame by frame, so as to determine whether the matching detection frames in different images belong to the same target.

[0164] Specifically, the detection frames belonging to the same object may be different detection frames whose image contents represent the same object.

[0165] Of course, there may be a situation where the number of detection frames between the previous frame and the next frame is different, resulting in the existence of unmatched detection frames. For a single detection frame that is not successfully matched in a continuous image, the detection frame can be regarded as a detection frame belonging to a target. For example, a target detection frame that only flashes in one frame. It is also possible not to add the detection frame to any detection frame set. Due to the high frequency of video data acquisition, the movement of the target will usually be captured by at least several consecutive frames. If it is a detection frame that only flashes in one frame, it can usually be considered an error result.

[0166] Optionally, based on the acquired detection frame, using a preset target tracking algorithm, determining one or more detection frame sets may include: determining the first frame image in the video data to be detected as the current image, and looping the following steps.

[0167] When there is a previous frame image in the current image, based on the first real detection frame obtained by S101 target detection in the previous frame image, the detection frame in the current image is predicted using preset parameters to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames.

[0168] For each second real detection frame obtained by S101 target detection in the current image, pair it with the obtained predicted detection frame. When it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add it to the detection frame set corresponding to the target.

[0169] Based on the pairing result, the preset parameters are updated. If the current image does not have a next frame image, the loop ends; if the current image does have a next frame image, the next frame image is determined as the current image.

[0170] Optionally, when there is no previous frame image for the current image, the next frame image of the current image may be directly determined as the current image.

[0171] Optionally, the preset parameters may include parameters of a Kalman filter. Correspondingly, predicting the detection frame in the current image using the preset parameters may include: predicting the detection frame in the current image using a Kalman filter.

[0172] Among them, the commodity tracking can adopt a fast tracking algorithm based on Kalman filtering. Through Kalman filtering, the commodity position and moving speed can be predicted. The specific updating of preset parameters, that is, updating the Kalman filter parameters, can include updating the moving speed used in the next prediction based on the movement between the detection frames of the same target in the real video data.

[0173] Regarding pairing, this method flow does not limit the specific pairing method.

[0174] Optionally, pairing may be performed based on the degree of overlap between the second real detection frame and the predicted detection frame. Specifically, for any second real detection frame, a predicted detection frame with the highest degree of overlap may be selected for pairing.

[0175] Optionally, the Hungarian algorithm may be used for pairing.

[0176] In order to improve the accuracy of pairing and target tracking, optionally, detection frame image features may be further added for pairing. Specifically, pairing may be performed based on the overlap between detection frames and the similarity of detection frame image features.

[0177] Optionally, the overlap between the comprehensive detection frames and the similarity of the detection frame image features may be considered, and for any second real detection frame, the predicted detection frame with the highest comprehensive result is selected for pairing.

[0178] Among them, the detection frame image features include, for example, color features, edge features, shape features, etc.

[0179] For example, for any second real detection frame, when there are two predicted detection frames with a high degree of detection frame overlap, if the image features in the second real detection frame are the same as the image features of one of the predicted detection frames, both are red features, then they most likely belong to the same target, and the predicted detection frame can be selected for pairing.

[0180] Of course, the Hungarian algorithm can be used for pairing.

[0181] Optionally, pairing each second real detection frame obtained through target detection in the current image with the obtained predicted detection frame can include: calculating preset detection frame image features for the image contained in each second real detection frame obtained through target detection in the current image and the image contained in the obtained predicted detection frame; and using the Hungarian algorithm to pair the second real detection frame with the predicted detection frame according to the similarity of the preset detection frame image features and the overlap of the detection frames.

[0182] Optionally, the similarity of the specific preset detection frame image features may be determined based on the cosine distance of the preset detection frame image feature vectors.

[0183] Among them, in order to reduce the amount of calculation and improve the extraction efficiency of the detection frame image features, thereby improving the target detection efficiency, efficient image features can be selected, which can effectively distinguish different targets and have low computational complexity and can be quickly extracted and processed in embedded systems.

[0184] For example, color features can be quickly determined based on the distribution of pixel values ​​in the image contained in the detection box, without the need for complex calculations of neural networks or other models.

[0185] Therefore, optionally, the computational complexity of the preset detection frame image features may be less than a preset complexity threshold. Optionally, the preset detection frame image features may include at least one of the following: color features, edge features, shape features, texture features, directional gradient histogram features, etc.

[0186] Optionally, the predicted detection box may be obtained by predicting each first real detection box in the previous frame of the image.

[0187] Optionally, if it is determined that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, the second real detection frame can be added to the detection frame set corresponding to the target. If the detection frame set corresponding to the target does not exist, the second real detection frame can be directly created and then added.

[0188] Specifically, the second real detection frame and the first real detection frame corresponding to the predicted detection frame are added to the detection frame set corresponding to the target.

[0189] It should be noted that, as the loop is executed, the detection frames of multiple consecutive frames can be paired, so that the detection frames belonging to the same target can be determined and added to the detection frame set corresponding to the target.

[0190] Optionally, since only the first frame image does not have a previous frame image for the video data to be detected, the next frame image of the first frame image in the video data to be detected can be optionally determined as the current image, and then the above-mentioned loop steps are performed.

[0191] Optionally, based on the acquired detection frame, one or more detection frame sets are determined using a preset target tracking algorithm, which may include: determining the next frame image of the first frame image in the video data to be detected as the current image, and looping the following steps: based on the first real detection frame obtained by target detection in the previous frame image, predicting the detection frame in the current image using preset parameters to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames; for each second real detection frame obtained by target detection in the current image, pairing it with the obtained predicted detection frame, and when it is determined that the second real detection frame is successfully paired with a predicted detection frame, determining that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and adding it to the detection frame set corresponding to the target; based on the pairing result, updating the above preset parameters; when there is no next frame image in the current image, ending the loop; when there is a next frame image in the current image, determining the next frame image as the current image.

[0192] For ease of understanding, the embodiment of the present invention also provides a flow chart of a target tracking method, such as Figure 2 shown. Figure 2 It is a flowchart of a target tracking method according to an embodiment of the present invention.

[0193] The following steps are included.

[0194] S201: Determine the next frame image of the first frame image in the video data to be detected as the current image.

[0195] S202: Based on a first real detection frame obtained by target detection in a previous frame image, a detection frame in a current image is predicted using preset parameters to obtain a predicted detection frame.

[0196] Among them, different predicted detection boxes correspond to different first real detection boxes.

[0197] S203: For each second real detection frame obtained through target detection in the current image, pair it with the obtained predicted detection frame. When it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add the second real detection frame to the detection frame set corresponding to the target.

[0198] S204: Based on the pairing result, update the above preset parameters.

[0199] S205: Determine whether there is a next frame image for the current image. If there is no next frame image for the current image, end this process. If there is a next frame image for the current image, execute S206.

[0200] S206: Determine the next frame image as the current image and execute S202.

[0201] For ease of understanding, the present invention also provides a specific embodiment.

[0202] Among them, the detected target can be a commodity, and the commodity tracking can adopt a fast tracking algorithm based on Kalman filtering, and a mechanism is specially designed to efficiently extract image features to improve tracking stability (tracking stability refers to the ability to continuously and stably track the same commodity to avoid loss as much as possible).

[0203] Tracking is performed iteratively. The specific process of processing a frame of video image is as follows:

[0204] First, Kalman filtering is used to predict the location and speed of the product to obtain tracks.

[0205] Then, the Hungarian algorithm is used to match the predicted tracks with the product detection results in the current frame video image (image feature matching and area IOU matching) to obtain (track, detection) paired data.

[0206] Finally, the Kalman filter parameters are updated using the detection results of the products that match the tracks.

[0207] The following are some examples:

[0208] Frame 0: The detector detects 3 detections. Currently, there are no tracks. These 3 detections are initialized as tracks.

[0209] Frame 1: The detector detects three more detections. For the tracks in Frame 0, we first predict new tracks, then use the Hungarian algorithm to match the new tracks with the detections to get (track, detection) matching pairs, and finally use the detection in each pair to update the corresponding track.

[0210] Then the loop is executed until the last frame of video data.

[0211] The specific method of extracting image features to improve tracking stability is as follows.

[0212] First, efficient image features (such as color features and HOG features) are selected based on feature engineering methods. The standard for efficient feature selection is that it can effectively distinguish different products and has low computational complexity so that it can be quickly extracted and processed in embedded systems.

[0213] Then, the track and detection are matched based on the image features and the regional IOU features, and the image feature matching uses the cosine distance of the feature vector.

[0214] S103: determining the image quality of the image contained in the detection frame in each detection frame set; and screening out the detection frame sets whose image quality is greater than a preset quality.

[0215] Among them, the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set.

[0216] Optionally, the image quality analysis may mainly analyze image conditions such as target occlusion and target motion blur, which may reduce the accuracy of target classification.

[0217] For example, in the set of detection frames corresponding to the target, if the images contained in the detection frames are all blurred and blocked by other objects, the accuracy of subsequent target classification will usually be low.

[0218] The method flow does not limit the form of image quality. Optionally, the image quality may be determined in a graded manner. The higher the grade, the higher the image quality. Correspondingly, the preset quality may specifically be a preset quality grade.

[0219] The method flow does not limit the way of determining the image quality, as long as the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set.

[0220] Optionally, the image quality may be determined by a neural network model.

[0221] Optionally, the image quality may be determined by a classification prediction method, a regression prediction method, or a combination of classification prediction and regression prediction methods.

[0222] The classification of image quality may be determined by grading.

[0223] Optionally, determining the image quality for the image contained in the detection frame in each detection frame set may include: inputting the image contained in the detection frame in each detection frame set into a preset quality analysis model; the preset quality analysis model may be used to determine the image quality; the preset quality analysis model may include a regression prediction branch and a classification prediction branch of the image quality; for any detection frame set, the image quality is determined by integrating the outputs of the regression prediction branch and the classification prediction branch in the preset quality analysis model.

[0224] This embodiment can improve the precision, accuracy and stability of image quality by integrating regression prediction and classification prediction.

[0225] Optionally, the preset quality analysis model may be a model with a small amount of calculation, and the calculation complexity may be less than a preset threshold, and may be a lightweight network based on mobilenetv2.

[0226] S104: Determine the category of the target corresponding to each detection frame set based on the image contained in the detection frame in each filtered detection frame set.

[0227] This method flow does not limit the method of specifically determining the category of the target.

[0228] Optionally, a target classification model may be used for determination, or analysis may be performed directly based on image features.

[0229] In an optional embodiment, the features of the target may be extracted from the image contained in the detection frame in the detection frame set, and then a comparison search may be performed from a preset category feature set to find category features that meet the requirements, thereby determining the category of the target.

[0230] In this embodiment, since the preset category feature set can be updated, the identifiable categories and the specific features of the categories can be adjusted more flexibly.

[0231] For example, for a newly added category, the corresponding category features can be directly added to the preset category feature set to determine the target belonging to the category. For a category that needs to update features, for example, the outer packaging of bottled milk has changed, the features of the bottled milk category in the preset category feature set can be directly replaced.

[0232] This embodiment can improve the flexibility of target classification by comparing the features of the target with a preset category feature set.

[0233] Therefore, optionally, determining the category of the target corresponding to the detection frame set based on the image contained in the detection frame in each filtered detection frame set may include: extracting features of the target based on the image contained in the detection frame in each filtered detection frame set, comparing the features of the target with features in a preset category feature set, and determining the category corresponding to the features whose similarity is greater than a preset similarity threshold as the category of the target corresponding to the detection frame set.

[0234] Optionally, the preset category feature set may include correspondences between several different categories and different features.

[0235] Alternatively, feature similarity can be calculated by the cosine distance of feature vectors.

[0236] This embodiment does not limit the feature extraction method of a specific target. Optionally, the feature extraction can be performed through a neural network model. Specifically, the hidden layer of the trained neural network model can be used to determine the hidden layer output as the feature of the target.

[0237] In an optional embodiment, during the target detection in S101, the target may be classified to determine a first category. Specifically, a general category may be determined, and a subdivided category of the target, that is, a second category, may be further determined in S104.

[0238] Since the first category of the target can be determined in S101, for a target of a certain first category, the possibility of other first categories can be excluded, and the second category of the target can be determined directly from the subcategories of the first category.

[0239] In this way, the possible second category range can be narrowed down for the targets of the determined first category, thereby saving the amount of calculation and improving the efficiency of target detection.

[0240] Therefore, optionally, performing target detection on the video data to be detected may include: performing detection on a first category of targets in the video data to be detected.

[0241] The preset classification feature set may include: features of the second category; the second category may be a sub-category of the first category.

[0242] Correspondingly, optionally, feature extraction models corresponding to each first category may be pre-set.

[0243] Extracting features of the target based on the image contained in the detection frame in each filtered detection frame set may include: determining, for each filtered detection frame set, a first category of the target corresponding to the detection frame set; and extracting features of the target for the image contained in the detection frame in each filtered detection frame set using a feature extraction model corresponding to the determined first category.

[0244] Among them, the feature extraction model corresponding to each first category can be specifically used to extract features for the corresponding first category target.

[0245] This embodiment does not limit the form of the feature extraction model. Optionally, the feature extraction model can be a model with a small amount of calculation, and the calculation complexity can be less than a preset threshold. Specifically, it can be a lightweight network based on mobilenetv2.

[0246] This embodiment also does not specifically limit the training method of the feature extraction model corresponding to each first category. Optionally, for multiple image samples under a single first category, a neural network model can be trained using the subdivision categories of the first category as labels, and the hidden layer of the trained neural network model can be output as the feature of the extracted target.

[0247] Specifically, the image contained in the detection box in each filtered detection box set is input into the trained neural network model, and the output of the hidden layer is determined as the feature of the target.

[0248] Optionally, since the first category to which the target belongs has been determined, when specifically comparing the features of the target with the features in a preset category feature set, only the subdivided category features included in the first category to which the target belongs in the preset category feature set may be compared, thereby improving the comparison efficiency by narrowing the scope.

[0249] It should be noted that, in an optional embodiment, during target detection in S101, what is usually output is the probability that the target belongs to each first category, and the first category to which the target belongs is determined by the probability.

[0250] Therefore, based on the probability that the target belongs to each first category, the feature extraction model to be used can be determined. Specifically, the feature extraction model corresponding to the first category with the highest output probability can be used, or the feature extraction model corresponding to the first category with an output probability greater than a threshold can be used.

[0251] Optionally, during target detection in S101, the first category to which the target belongs may not be determined based on the probability, or the first category whose probability is not greater than a threshold may be output.

[0252] In other words, during target detection, it may be difficult to determine the first category of the target.

[0253] Correspondingly, optionally, a preset feature extraction model may be used to extract features. The preset feature extraction model may be used to extract features for each first category or each second category of targets.

[0254] This embodiment does not limit the form of the preset feature extraction model. Optionally, the feature extraction model can be a model with a small amount of calculation, and the calculation complexity can be less than a preset threshold. Specifically, it can be a lightweight network based on mobilenetv2 or a model based on resnet18. This embodiment can reduce the amount of calculation and improve the efficiency of feature extraction.

[0255] This embodiment also does not specifically limit the training method of the preset feature extraction model. Optionally, a neural network model can be trained based on a set of image samples including all second category labels, and the hidden layer output of the trained neural network model can be used as the feature of the extracted target.

[0256] For ease of understanding, the present invention also provides a flow chart of a feature extraction method. Figure 3 As shown, Figure 3 It is a flowchart of a feature extraction method according to an embodiment of the present invention.

[0257] Specifically, the following steps may be included.

[0258] S301: Determine whether the first category of the target is determined. If the first category of the target is determined, execute S302; if the first category of the target is not determined, execute S304.

[0259] S302: Using the determined feature extraction model corresponding to the first category, extract features of the target for an image contained in a detection frame in a detection frame set corresponding to the target.

[0260] S303: Compare the extracted features with the subdivided category features included in the determined first category, and determine the subdivided category corresponding to the features having a similarity greater than a preset similarity threshold as the category of the target.

[0261] S304: Using a preset feature extraction model, extract features of the target from an image contained in a detection frame in a detection frame set corresponding to the target.

[0262] S305: Compare the extracted features with all subdivided category features, and determine the subdivided category corresponding to the features with similarity greater than a preset similarity threshold as the category of the target.

[0263] For ease of understanding, the present invention also provides a specific embodiment.

[0264] In the scenario of smart refrigerators, four product feature extraction models are specially designed to improve the accuracy and speed of product feature extraction.

[0265] These include three lightweight models based on MobileNetV2 for bottled, boxed and bagged products, which are used to further determine the product subcategories; and one model based on ResNet18 for all product subcategories.

[0266] Through object detection, the category confidence of the product detection box can be determined, that is, the confidence of the bottled category, boxed category, and bagged category.

[0267] When the confidence of the bottled category is greater than a preset threshold, such as 0.6, the feature extraction model for bottled products is used to extract product features, and the features are compared with the subdivided category features of the bottled products to obtain a comparison result. Specifically, the subdivided category corresponding to the feature with a similarity greater than a preset similarity threshold is determined as the subdivided category of the bottled products.

[0268] When the confidence of the bagged category is greater than a preset threshold, such as 0.6, the product features are extracted using the feature extraction model for bagged category products, and compared with the subdivided category features of bagged category products to obtain a comparison result. Specifically, the subdivided category corresponding to the feature with a similarity greater than a preset similarity threshold is determined as the subdivided category of the bagged category products.

[0269] When the confidence of the boxed category is greater than a preset threshold, such as 0.6, the product features are extracted using the feature extraction model for the boxed category products, and compared with the subdivided category features of the boxed category products to obtain a comparison result. Specifically, the subdivided category corresponding to the feature with a similarity greater than a preset similarity threshold is determined as the subdivided category of the boxed category products.

[0270] In other cases, it may be considered that the product is not identified as being in a bottle, box or bag.

[0271] Therefore, a feature extraction model for all product subdivision categories can be used to extract product features and compare them with all subdivision category features to obtain a comparison result. Specifically, the subdivision category corresponding to the feature with a similarity greater than a preset similarity threshold can be determined as the product subdivision category.

[0272] In an optional embodiment, for feature comparison, the speed of feature comparison can be increased by reducing the precision without reducing the accuracy of feature comparison.

[0273] Therefore, optionally, the features in the preset category feature set can be set as features of preset accuracy, and the accuracy of the features of the extracted target can be reduced to the preset accuracy, so that direct comparison can be performed, reducing the amount of calculation, increasing the comparison speed, and improving the efficiency of target detection.

[0274] In a specific embodiment, the accuracy of the conventional output feature vector of the commodity feature extraction is FP32, and the feature comparison is also performed based on the FP32 feature vector.

[0275] In this method, features with FP16, INT16, and INT8 precision can be extracted for comparison, which can effectively improve the comparison speed.

[0276] In the process of this method, a large number of neural network models can be used, for example, target detection models, image quality analysis models and target classification models.

[0277] In order to further reduce the amount of calculation and improve the efficiency of target detection, the neural network model itself can be optimized.

[0278] Optionally, model quantization or model pruning can be performed on any neural network model.

[0279] This method does not limit the specific model quantization method or model pruning method.

[0280] Optionally, automatic mixed precision quantization may be performed for any neural network model, wherein any neural network model may be any neural network model used in the process of the present method.

[0281] Optionally, among the first neural network model for performing target detection on the video data to be detected, the second neural network model for determining the image quality, and the third neural network model for determining the category of the target corresponding to the detection frame set, at least one neural network model is obtained by the following quantization method.

[0282] The initial neural network model trained with the first parameter accuracy is quantized to the second parameter accuracy to obtain an intermediate neural network model; the second parameter accuracy is lower than the first parameter accuracy.

[0283] The following steps are executed in a loop until the accuracy and computing speed of the current intermediate neural network model meet the preset quantization requirements: determine the accuracy and computing speed of the current intermediate neural network model; determine the output error of each layer between the initial neural network model and the current intermediate neural network model; select the layer with the largest output error to improve parameter accuracy, and obtain a new intermediate neural network model as the current intermediate neural network model.

[0284] Among them, the lower the parameter precision of the model, the faster the calculation speed, and accordingly, the accuracy may decrease.

[0285] The preset quantitative requirement may specifically be a requirement for a comprehensive result of accuracy and calculation speed. Alternatively, the calculation speed may be greater than a preset speed when the accuracy is not less than a preset accuracy, thereby improving the efficiency of target detection.

[0286] Optionally, the current intermediate neural network model obtained after the cycle ends can be used in the process of this method.

[0287] For ease of understanding, the present invention also provides a flow chart of a model quantization method. Figure 4 As shown, Figure 4 It is a flowchart of a model quantization method according to an embodiment of the present invention.

[0288] The method may include the following steps.

[0289] S401: quantize the initial neural network model trained with the first parameter precision to the second parameter precision to obtain an intermediate neural network model, wherein the first parameter precision may be FP32 and the second parameter precision may be INT8.

[0290] S402: Determine the accuracy and calculation speed of the current intermediate neural network model.

[0291] S403: Determine whether the accuracy and calculation speed of the current intermediate neural network model meet the preset quantization requirements. If the preset quantization requirements are met, end this process. If the preset quantization requirements are not met, execute S404.

[0292] S404: Determine the output error of each layer between the initial neural network model and the current intermediate neural network model.

[0293] S405: Select the layer with the largest output error to improve parameter accuracy, obtain a new intermediate neural network model as the current intermediate neural network model, and execute S402.

[0294] Corresponding to the above method embodiment, the embodiment of the present invention also provides a device embodiment.

[0295] like Figure 5 As shown, Figure 5 It is a structural schematic diagram of a dynamic target analysis device according to an embodiment of the present invention.

[0296] The apparatus may include the following units.

[0297] The target detection unit 501 is used to perform target detection on the video data to be detected and obtain the target detection frame of each frame image.

[0298] The target tracking unit 502 is used to determine one or more detection frame sets based on the acquired detection frame and using a preset target tracking algorithm; wherein each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets.

[0299] The screening unit 503 is used to determine the image quality of the image contained in the detection frame in each detection frame set; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; and the detection frame set with image quality greater than the preset quality is screened out.

[0300] The classification unit 504 is used to determine the category of the target corresponding to each detection frame set based on the image contained in the detection frame in each filtered detection frame set.

[0301] Optionally, the classification unit 504 is used to: extract features of the target based on the image contained in the detection frame in each filtered detection frame set, compare the features of the target with features in a preset category feature set, and determine the category corresponding to the features whose similarity is greater than a preset similarity threshold as the category of the target corresponding to the detection frame set.

[0302] Optionally, the target detection unit 501 is used to: detect a first category of the target in the video data to be detected; the preset classification feature set includes: features of the second category; the second category is a subdivision category of the first category; the device is pre-set with feature extraction models corresponding to each first category; the classification unit 504 is used to: determine the first category of the target corresponding to each filtered detection frame set; use the feature extraction model corresponding to the determined first category to extract the features of the target for the image contained in the detection frame in each filtered detection frame set.

[0303] Optionally, the target detection unit 501 is used to: input the video data to be detected into a preset target detection model, the preset target detection model is used to determine the third category of the target; the preset target detection model includes a detection box prediction branch corresponding to each third category; when the third category of any target is determined, the detection box output by the detection box prediction branch corresponding to the third category is determined as the detection box of the target.

[0304] Optionally, the target detection unit 501 is used to: input the video data to be detected into a preset target detection model, the preset target detection model is used to determine the detection frame position, size and rotation angle of the target; the preset target detection model includes a regression prediction branch and a classification prediction branch of the detection frame rotation angle; for any target, the outputs of the regression prediction branch and the classification prediction branch in the preset target detection model are integrated to determine the comprehensive result of the rotation angle.

[0305] Optionally, the target tracking unit 502 is used to: determine the next frame image of the first frame image in the video data to be detected as the current image, and loop the following steps: based on the first real detection frame obtained by target detection in the previous frame image, use preset parameters to predict the detection frame in the current image to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames; for each second real detection frame obtained by target detection in the current image, pair it with the obtained predicted detection frame, and when it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add it to the detection frame set corresponding to the target; based on the pairing result, update the preset parameters; when there is no next frame image in the current image, end the loop; when there is a next frame image in the current image, determine the next frame image as the current image.

[0306] Optionally, the preset parameters include parameters of a Kalman filter; a target tracking unit 502, configured to: predict a detection frame in a current image using a Kalman filter; calculate preset detection frame image features for an image contained in each second real detection frame obtained by target detection in the current image, and an image contained in the obtained predicted detection frame; the computational complexity of the preset detection frame image features is less than a preset complexity threshold; and pair the second real detection frame with the predicted detection frame using a Hungarian algorithm based on the similarity of the preset detection frame image features and the overlap of the detection frames.

[0307] Optionally, the screening unit 503 is used to: input the image contained in the detection box in each detection box set into a preset quality analysis model; the preset quality analysis model is used to determine the image quality; the preset quality analysis model includes a regression prediction branch and a classification prediction branch of the image quality; for any detection box set, the outputs of the regression prediction branch and the classification prediction branch in the preset quality analysis model are integrated to determine the image quality.

[0308] Optionally, among the first neural network model for performing target detection on the video data to be detected, the second neural network model for determining the image quality, and the third neural network model for determining the category of the target corresponding to the detection frame set, at least one neural network model is obtained by the following quantization method:

[0309] The initial neural network model trained with the first parameter accuracy is quantized to the second parameter accuracy to obtain an intermediate neural network model; the second parameter accuracy is lower than the first parameter accuracy;

[0310] The following steps are executed repeatedly until the accuracy and computing speed of the current intermediate neural network model meet the preset quantization requirements:

[0311] Determine the accuracy and computational speed of the current intermediate neural network model;

[0312] Determine the output error of each layer between the initial neural network model and the current intermediate neural network model;

[0313] The layer with the largest output error is selected to improve parameter accuracy, and a new intermediate neural network model is obtained as the current intermediate neural network model.

[0314] An embodiment of the present invention further provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein any of the above method embodiments is implemented when the processor executes the program.

[0315] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned method embodiments.

[0316] Figure 6 1 is a schematic diagram of the hardware structure of a computer device configured with a method according to an embodiment of the present invention, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other in communication within the device through the bus 1050.

[0317] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0318] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solution provided by the embodiment of the present invention is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0319] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0320] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0321] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0322] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0323] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any of the above method embodiments is implemented.

[0324] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements any of the above method embodiments when executed by a processor.

[0325] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0326] It can be known from the description of the above implementation mode that a person skilled in the art can clearly understand that the embodiment of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiment of the present invention can be essentially or in other words, the part that makes the contribution can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment of the present invention.

[0327] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0328] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present invention. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0329] The above is only a specific implementation of the embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present invention. These improvements and modifications should also be regarded as protection for the embodiment of the present invention.

[0330] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0331] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0332] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A dynamic target analysis method, characterized in that: include: Perform target detection on the video data to be detected and obtain the target detection frame of each frame image; Based on the acquired detection frames, one or more detection frame sets are determined using a preset target tracking algorithm; wherein each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets; For the image contained in the detection frame in each detection frame set, determine the image quality; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; and filter out the detection frame set whose image quality is greater than the preset quality; Based on the images contained in the detection frames in each of the filtered detection frame sets, the category of the target corresponding to the detection frame set is determined.

2. The method according to claim 1, characterized in that The determining, based on the image contained in the detection frame in each of the filtered detection frame sets, the category of the target corresponding to the detection frame set includes: Based on the image contained in the detection frame in each filtered detection frame set, the features of the target are extracted, and the features of the target are compared with the features in the preset category feature set. The category corresponding to the features whose similarity is greater than the preset similarity threshold is determined as the category of the target corresponding to the detection frame set.

3. The method according to claim 2, characterized in that The target detection for the video data to be detected includes: Detecting a first category of an object in the video data to be detected; The preset category feature set includes: features of a second category; the second category is a sub-category of the first category; The method further comprises: presetting a feature extraction model corresponding to each first category; The extracting the features of the target based on the image contained in the detection frame in each of the filtered detection frame sets includes: For each filtered detection frame set, determine the first category of the target corresponding to the detection frame set; The determined feature extraction model corresponding to the first category is used to extract target features for the image contained in the detection frame in each of the screened detection frame sets.

4. The method according to claim 1, characterized in that: The target detection for the video data to be detected includes: Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine a third category of the target; the preset target detection model includes a detection box prediction branch corresponding to each third category; When the third category of any target is determined, the detection box output by the detection box prediction branch corresponding to the third category is determined as the detection box of the target.

5. The method according to claim 1, characterized in that The target detection for the video data to be detected includes: Inputting the video data to be detected into a preset target detection model, wherein the preset target detection model is used to determine the detection frame position, size and rotation angle of the target; The preset target detection model includes a regression prediction branch and a classification prediction branch of the detection frame rotation angle; For any target, the outputs of the regression prediction branch and the classification prediction branch in the preset target detection model are integrated to determine the comprehensive result of the rotation angle.

6. The method according to claim 1, characterized in that The step of determining one or more detection frame sets based on the acquired detection frame and using a preset target tracking algorithm includes: The next frame image of the first frame image in the video data to be detected is determined as the current image, and the following steps are performed in a loop: Based on the first real detection frame obtained by target detection in the previous frame image, the detection frame in the current image is predicted using preset parameters to obtain a predicted detection frame; different predicted detection frames correspond to different first real detection frames; For each second real detection frame obtained by target detection in the current image, pair it with the obtained predicted detection frame. When it is determined that the second real detection frame is successfully paired with a predicted detection frame, determine that the second real detection frame and the first real detection frame corresponding to the predicted detection frame belong to the same target, and add it to the detection frame set corresponding to the target; Based on the pairing result, updating the preset parameters; If the next frame does not exist in the current image, the loop ends; When the current image has a next frame image, the next frame image is determined as the current image.

7. The method according to claim 6, characterized in that The preset parameters include: Kalman filter parameters; The predicting the detection frame in the current image by using preset parameters includes: predicting the detection frame in the current image by using Kalman filtering; The step of pairing each second real detection frame obtained through target detection in the current image with the obtained predicted detection frame includes: For each image contained in the second real detection frame obtained by target detection in the current image, and the image contained in the obtained predicted detection frame, a preset detection frame image feature is calculated; the calculation complexity of the preset detection frame image feature is less than a preset complexity threshold; Using the Hungarian algorithm, the second real detection frame is paired with the predicted detection frame according to the similarity of the preset detection frame image features and the overlap of the detection frames.

8. The method according to claim 1, characterized in that The determining the image quality of the image contained in the detection frame in each detection frame set includes: Inputting the image contained in the detection frame in each detection frame set into a preset quality analysis model; the preset quality analysis model is used to determine the image quality; the preset quality analysis model includes a regression prediction branch and a classification prediction branch of the image quality; For any set of detection frames, the image quality is determined by combining the outputs of the regression prediction branch and the classification prediction branch in the preset quality analysis model.

9. The method according to claim 1, characterized in that: Among the first neural network model for performing target detection on the video data to be detected, the second neural network model for determining the image quality, and the third neural network model for determining the category of the target corresponding to the detection frame set, at least one neural network model is obtained by the following quantization method: The initial neural network model trained with the first parameter accuracy is quantized to the second parameter accuracy to obtain an intermediate neural network model; the second parameter accuracy is lower than the first parameter accuracy; The following steps are executed repeatedly until the accuracy and computing speed of the current intermediate neural network model meet the preset quantization requirements: Determine the accuracy and computational speed of the current intermediate neural network model; Determining the output error of each layer between the initial neural network model and the current intermediate neural network model; The layer with the largest output error is selected to improve parameter accuracy, and a new intermediate neural network model is obtained as the current intermediate neural network model.

10. A dynamic target analysis device, characterized in that: include: The target detection unit is used to perform target detection on the video data to be detected and obtain the target detection frame of each frame image; A target tracking unit, configured to determine one or more detection frame sets based on the acquired detection frame and using a preset target tracking algorithm; wherein each detection frame set includes detection frames belonging to the same target between different images, and different detection frame sets correspond to different targets; A screening unit is used to determine the image quality of the image contained in the detection frame in each detection frame set; the image quality is positively correlated with the classification accuracy of the target corresponding to the detection frame set; and screen out the detection frame set whose image quality is greater than a preset quality; The classification unit is used to determine the category of the target corresponding to each detection frame set based on the image contained in the detection frame in each filtered detection frame set.

11. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that: The computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

Images