Target detection model training method, system, device and storage medium

Abstract

The present invention provides a method, system, device and storage medium for training a target detection model, the method comprising: a central processing unit receives video data collected by a video acquisition device, converts the received video data into multiple image frames, and divides the received video data into a training set and a test set; the image processing unit performs training annotation on the training set; the image processing unit inputs the image frames and their training annotations into a target detection model for training, the target detection model is a convolutional neural network model; the image processing unit performs test annotation on the test set; the image processing unit inputs multiple image frames in the test set into the target detection model to obtain target detection results; the central processing unit compares the test annotations with the target detection results to obtain the test evaluation results of the target detection model. The present invention reduces the consumption of computing resources of the central processing unit, improves the accuracy of target detection, and further improves the efficiency of the target detection system.

Description

Background Art

[0002] Image processing, especially target detection in image processing, has been widely used in intelligent fields such as monitoring and autonomous driving. In the target detection method, an algorithm is required to detect targets (such as people, vehicles, etc.) in the video captured by the camera. At present, the mainstream algorithm uses optical flow method or frame difference method, combined with the box drawing function to confirm the target area. However, the above method will inevitably consume a lot of computing resources of the central processing unit, thus causing the computing bottleneck of the entire detection system.

[0003] Therefore, how to optimize the target detection method to reduce the consumption of computing resources of the central processing unit and improve the accuracy of target detection and further improve the efficiency of the target detection system is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] In view of the problems in the prior art, the purpose of the present invention is to provide a target detection model training method, system, device and storage medium to reduce the consumption of computing resources of the central processing unit, improve the accuracy of target detection, and further improve the efficiency of the target detection system.

[0005] The present invention provides a method for training a target detection model, comprising the following steps:

[0006] The central processing unit receives video data collected by the video collection device, and converts the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set;

[0007] The image processing unit executes a pre-trained target detection model to perform training annotation on a plurality of image frames in the training set;

[0008] The image processing unit inputs a plurality of image frames in the training set and their training annotations into a target detection model for training, wherein the target detection model is a convolutional neural network model;

[0009] The image processing unit executes a pre-trained target detection model to perform test annotation on multiple image frames in the test set;

[0010] The image processing unit inputs a plurality of image frames in the test set into the target detection model to obtain a target detection result;

[0011] The central processing unit compares the test annotation and the target detection result to obtain a test evaluation result of the target detection model.

[0012] In some embodiments of the present application, the loss function of the target detection model is calculated based on the KL distance, the background target difference penalty term and the region intersection-and-union ratio, wherein the KL distance is the discreteness of the probability distribution of the training annotations and the probability distribution of the predicted output of the target detection model, the background target difference penalty term is the reciprocal of the difference between the average brightness of the target area of ​​the predicted output of the target detection model and the background area, and the region intersection-and-union ratio is the intersection-and-union ratio of the target area of ​​the predicted output of the target detection model and the target area of ​​the training annotation.

[0013] In some embodiments of the present application, the loss function L of the target detection model is calculated according to the following formula:

[0014] L=a*KLD+b*penalization+c*(1-IoU),

[0015] Among them, a, b, c are weights, and the sum of a, b, c is 1, KLD is the KL distance, penalization is the background target difference penalty term, and IoU is the region intersection over union ratio.

[0016] In some embodiments of the present application, the central processing unit simultaneously receives video data collected by multiple video acquisition devices, and the target detection model is used to perform target detection on the video data collected by the multiple video acquisition devices.

[0017] In some embodiments of the present application, the target detection model is trained for multiple iterations, and the pre-trained target detection model is the target detection model trained for the previous iteration.

[0018] According to another aspect of the present application, a target detection method is also provided, comprising:

[0019] The central processing unit receives video data collected by multiple video collection devices and converts the received video data into multiple image frames;

[0020] The image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model, and the target detection model is trained by the target detection model training method as described above;

[0021] The central processing unit obtains the target detection result output by the target detection model.

[0022] In some embodiments of the present application, the image processing unit inputting the multiple image frames converted by the central processing unit into the target detection model further includes:

[0023] Obtaining current load parameters of the central processing unit;

[0024] If the load parameter indicates that the central processing unit is in an idle state, the central processing unit performs target detection on a plurality of image frames converted by the central processing unit based on a frame difference algorithm;

[0025] If the load parameter indicates that the central processing unit is in a busy state, the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model.

[0026] According to another aspect of the present application, a target detection model training system is also provided, comprising:

[0027] A receiving and converting module, configured to receive video data collected by a video acquisition device by a central processing unit, and convert the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set;

[0028] A training and annotation module, configured to execute a pre-trained target detection model by an image processing unit to perform training and annotation on a plurality of image frames in the training set;

[0029] A training module, configured to input the plurality of image frames and their training annotations in the training set into a target detection model for training by the image processing unit, wherein the target detection model is a convolutional neural network model;

[0030] A test annotation module, configured to execute a pre-trained target detection model by the image processing unit to perform test annotation on a plurality of image frames in the test set;

[0031] An input module, configured to input a plurality of image frames in the test set into the target detection model by the image processing unit to obtain a target detection result;

[0032] A test evaluation module is used for the central processing unit to compare the test annotation and the target detection result to obtain a test evaluation result of the target detection model.

[0033] An embodiment of the present invention further provides an electronic device, including:

[0034] processor;

[0035] a memory storing executable instructions of the processor;

[0036] Wherein, the processor is configured to perform the steps of the target detection model training method by executing the executable instructions.

[0037] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the target detection model training method.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0039] The target detection model training method, target detection method, system, device and storage medium of the present invention have the following beneficial effects:

[0040] The present application uses a central processing unit and an image processing unit to respectively execute different steps in target detection model training and target detection, and makes the target detection model a convolutional neural network, so that the training process and the detection process of the target detection model are executed in the image processing unit. On the one hand, the computing power of the image processing unit is used to perform complex calculations, which reduces the consumption of computing resources of the central processing unit and further improves the overall target detection efficiency of the system; on the other hand, compared with frame difference and optical flow algorithms, the target detection accuracy of the convolutional neural network is higher, thereby also improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0042] Figure 1 is a flow chart of a target detection model training method according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of a target detection scenario according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a target detection model according to an embodiment of the present invention;

[0045] Figure 4 is a flow chart of a target detection method according to an embodiment of the present invention;

[0046] Figure 5 is a flow chart of selecting a central processing unit or an image processing unit to perform target detection according to an embodiment of the present invention;

[0047] Figure 6 is a schematic diagram of the structure of a target detection model training system according to an embodiment of the present invention;

[0048] Figure 7 is a schematic structural diagram of a target detection system according to an embodiment of the present invention;

[0049] Figure 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0050] Fig. 9 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0052] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0053] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined. Therefore, the actual execution order may be changed according to the actual situation.

[0054] like Figure 1 As shown, an embodiment of the present invention provides a target detection model training method, the method comprising the following steps:

[0055] Step S111: The central processing unit receives video data captured by the video capture device, and converts the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set.

[0056] Specifically, the reception of video data and the conversion of image frames are both performed by the central processing unit. The central processing unit can split the video data into continuous image frames at a set frame rate. The set frame rate can be, for example, 25 frames per second. The present application is not limited to this, and the set frame rate can be set according to the frame rate of the video acquisition device. The video acquisition device can be, for example, a camera.

[0057] Specifically, the central processing unit may save all image frames generated by each video data into the same folder to provide the same image frame reading path, thereby improving the image frame reading efficiency.

[0058] Specifically, 80% of the image frames (folders) can be divided into a training set, and 20% of the image frames (folders) can be divided into a test set. The training set is used to train the target detection model, and the test set is used to measure the performance of the model. The present application can also implement other distribution ratios of training sets and test sets, which is not limited to this.

[0059] Step S121: the image processing unit executes the pre-trained target detection model to perform training annotation on multiple image frames in the training set.

[0060] Specifically, the pre-trained target detection model can be a target detection model provided by other projects or a third party.

[0061] In some variations, since the target detection model can be updated through iterative training, the target detection model of the last update (last iteration) can be used as a pre-trained target detection model to improve the labeling accuracy of the pre-trained target detection model.

[0062] Furthermore, when there is no pre-trained target detection model, the training annotation (and the test annotation in the following steps) can be performed by the central processing unit. In this case, the training annotation (and the test annotation in the following steps) is processed manually, and the training annotation (and the test annotation in the following steps) of each image frame is saved through interaction with the central processing unit. After the target detection model is trained based on manual annotation, the trained target detection model can be used as a pre-trained target detection model.

[0063] In another variation, the pre-trained target detection model may have a different network structure from the target detection model. Furthermore, there may be multiple pre-trained target detection models, each of which has different sources (for example, including target detection models provided by the above-mentioned other projects or third parties, target detection models updated in the previous time, and pre-trained target detection models with different network structures from the target detection model). Thus, the training annotations of pre-trained target detection models from different sources can be combined to improve the accuracy of the training annotations. The combination method can be set as needed. For example, in some embodiments, training annotations and test annotations can be performed with different pre-trained target detection models, respectively, and according to the test evaluation results, the pre-trained target detection model with the best test evaluation results is used as the pre-trained target detection model for training annotation during the next update iteration.

[0064] Specifically, the above training annotation is to annotate the salient areas of the image frames. The specific annotation can be: for each image frame, construct a background image with the same size as the image frame (all pixels are 0), correspond the area of ​​the target (such as people, vehicles, etc.) on each image frame to the background image, perform brightness annotation (set the brightness to 1), and use the background image as the training annotation.

[0065] Step S122: The image processing unit inputs the multiple image frames in the training set and their training annotations into the target detection model for training, and the target detection model is a convolutional neural network model.

[0066] Specifically, the target detection model is made into a convolutional neural network model so that the target detection model can be trained and detected in the image processing unit.

[0067] Specifically, a group of image frames in the training set can be used as input image groups, and the corresponding training annotations can be used as corresponding annotations, which are input into the target detection model. After multiple rounds of iterations (which can be set rounds of iterations, or can be iterations to reach a set accuracy), a trained target detection model with better performance can be obtained. The iterative process can be based on a predefined target loss function, and the difference between the probability model predicted by the model and the true probability model can be measured by the target loss function. In each round of model iteration, the target loss function is continuously reduced by the gradient descent method, thereby achieving the purpose of model optimization.

[0068] Specifically, the loss function of the target detection model can be calculated based on the KL distance (Kullback-Leibler_divergence, KLD, used to measure the distance between two probability distributions), the background target difference penalty term and the area intersection and union ratio, wherein the KL distance is the discreteness of the probability distribution of the training annotation and the probability distribution of the predicted output of the target detection model, the background target difference penalty term is the reciprocal of the difference between the average brightness of the target area and the background area of ​​the predicted output of the target detection model (the greater the difference in the average brightness of the target area and the background area, the smaller the value of the background target difference penalty term), and the area intersection and union ratio is the intersection and union ratio of the target area of ​​the predicted output of the target detection model and the target area of ​​the training annotation.

[0069] Furthermore, the loss function L of the target detection model can be calculated according to the following formula:

[0070] L=a*KLD+b*penalization+c*(1-IoU),

[0071] Wherein, a, b, c are weights, and the sum of a, b, c is 1, KLD is the KL distance, penalization is the background target difference penalty term, and IoU is the region intersection over union ratio. Considering the contribution of each item to the loss function, a can be taken from 0.1 to 0.3, b can be taken from 0.4 to 0.5, and c can be taken from 0.3 to 0.4. In a preferred embodiment, a is 0.2, b is 0.45, and c is 0.35.

[0072] Among them, KLD can be calculated according to the following formula:

[0073]

[0074] Wherein, KDL(p||q) is the KL distance, p(x) is the probability distribution of the training annotation, q(x) is the probability distribution of the predicted output of the target detection model, X is the target set, and x is the detected target.

[0075] Step S123: The image processing unit executes the pre-trained target detection model to perform test annotation on multiple image frames in the test set.

[0076] Specifically, the implementation of the test annotation can be the same as the implementation of the training annotation to improve the annotation efficiency of the image processing unit and the accuracy of the test evaluation. The test annotation can be performed simultaneously with the training annotation.

[0077] Step S124: the image processing unit inputs the multiple image frames in the test set into the target detection model to obtain a target detection result.

[0078] Step S112: The central processing unit compares the test annotation and the target detection result to obtain a test evaluation result of the target detection model.

[0079] Specifically, the test evaluation result is used to evaluate the similarity between the target detection result and the test annotation. The test evaluation result can be displayed in different forms of expression. In some embodiments, the similarity between the target detection result and the test annotation can be displayed as the accuracy of the target detection model. In some other embodiments, the probability distribution of the target detection result and the probability distribution of the test annotation can also be displayed to obtain more detailed test evaluation information. The present application can achieve more variations, which will not be repeated here.

[0080] The present application uses a central processing unit and an image processing unit to respectively execute different steps in target detection model training and target detection, and makes the target detection model a convolutional neural network, so that the training process and the detection process of the target detection model are executed in the image processing unit. On the one hand, the computing power of the image processing unit is used to perform complex calculations, which reduces the consumption of computing resources of the central processing unit and further improves the overall target detection efficiency of the system; on the other hand, compared with frame difference and optical flow algorithms, the target detection accuracy of the convolutional neural network is higher, thereby also improving the accuracy of target detection.

[0081] See below Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a target detection scenario according to an embodiment of the present invention. Figure 2As shown, in this embodiment, the central processing unit 10 needs to communicate with multiple video acquisition modules 30A-30N to perform the acquisition of video data of the video acquisition module and the control of the video acquisition module 30A-30N (such as rotation angle, screen zooming, etc.). It is precisely because the central processing unit 10 needs to control multiple video acquisition modules 30A-30N that the central processing unit 10 often has the problem of excessive load. Therefore, if there is still a central processing unit 10 to perform target detection model training and target detection, the load of the central processing unit will be further increased, affecting the central processing unit 10's control of the video acquisition module 30A-30N, thereby affecting target detection and target tracking. Further, in some specific implementations, when applied to the field of autonomous driving, especially in the autonomous driving of ports, the central processing unit 10 will also communicate with vehicle-mounted equipment, autonomous driving servers, etc., or the central processing unit 10 can also be used as an autonomous driving server. After obtaining the target detection result, the central processing unit 10 will communicate with the vehicle-mounted equipment (and the autonomous driving server) to facilitate path planning, obstacle avoidance, etc. of autonomous driving. It can be seen that the computing speed and communication speed of the central processing unit 10 play a vital role in the safety of autonomous driving. Therefore, the image processing unit 20 is used to share the complex calculation of image processing for the central processing unit 10 to reduce the computing load of the central processing unit 10, which not only enables the central processing unit 10 to control multiple video acquisition modules 30A-30N at the same time, but also has a higher response speed to send the target detection information to the vehicle-mounted device (and the autonomous driving server) for communication, thereby improving the safety of autonomous driving.

[0082] See below Figure 3 , Figure 3 Schematic diagram of an object detection model of an embodiment of the present invention. In this embodiment, the object detection model may be TASED-Net (Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection).

[0083] The input of the target detection model is a plurality of continuous image frames, for example, it can be T consecutive image frames with a height of H and a width of W. The plurality of continuous image frames are input from the first convolution layer, and the output of the first convolution layer is connected to the maximum pooling layer MP1 (the convolution kernel is 1*1*1, and the step size is 1,2,2) and the maximum pooling layer MP4 in the auxiliary pooling (the pooling kernel is 4*1*1, and the step size is 4,1,1). The output of the maximum pooling layer MP1 is connected to the hybrid convolution layer MC1, and the output of the hybrid convolution layer MC1 is connected to the maximum pooling layer MP2 (the convolution kernel is 3*3*3, and the step size is 2,2,2) and the maximum pooling layer MP6 in the auxiliary pooling (the pooling kernel is 4*1*1, and the step size is 4,1,1). The output of the maximum pooling layer MP2 is connected to the hybrid convolution layer MC2, and the output of the hybrid convolution layer MC2 is connected to the second convolution layer and the maximum pooling layer MP8 in the auxiliary pooling (the pooling kernel is 2*1*1, and the step size is 2,1,1). The output of the second convolution layer is connected to the deconvolution layer CT1 (the convolution kernel is 1*3*3). The output of the deconvolution layer CT1 is connected to the upper pooling layer 1 (the pooling kernel is 1*2*2, and the step size is 1,2,2). The input of the upper pooling layer 1 also includes the output after the maximum pooling through the maximum pooling layer MP8 output to the maximum pooling layer MP9 (the pooling kernel is 1*2*2, and the step size is 1,2,2). The output of the upper pooling layer 1 is connected to the deconvolution layer CT2 (the convolution kernel is 1*3*3). The output of the deconvolution layer CT2 is connected to the upper pooling layer 2 (the pooling kernel is 1*3*3, and the step size is 1,2,2). The input of the upper pooling layer 2 also includes the output after the maximum pooling through the maximum pooling layer MP6 output to the maximum pooling layer MP7 (the pooling kernel is 1*3*3, and the step size is 1,2,2). The output of the upper pooling layer 2 is connected to the deconvolution layer CT3 (the convolution kernel is 1*3*3). The output of the deconvolution layer CT3 is connected to the upper pooling layer 3 (pooling kernel is 1*3*3, step size is 1,2,2). The input of the upper pooling layer 3 also includes the output after maximum pooling through the maximum pooling layer MP4 output to the maximum pooling layer MP5 (pooling kernel is 1*3*3, step size is 1,2,2). The output of the upper pooling layer 3 is connected to the deconvolution layer CT4 (convolution kernel is 1*4*4, step size is 1,2,2). The output of the deconvolution layer CT4 is connected to the third convolution layer, the output of the third convolution layer is connected to the deconvolution layer CT5, the output of the deconvolution layer CT5 is connected to the fourth convolution layer, and the output of the fourth convolution layer is the output of the target detection model.

[0084] In this embodiment, Figure 3The first row of layers in the figure are encoding layers, the third row of layers are decoding layers, and the layers in the dotted box in the second row are used for auxiliary pooling. In this embodiment, the target detection model extracts information from each image frame through convolution and deconvolution in 3D dimensions, and uses auxiliary pooling to reconstruct time series information during the deconvolution process. After each layer of convolution, there will be a batch normalization layer and a rectified linear layer (Relu) as a transition, and the final output layer can end with an activation function (Sigmoid) and output to a salient background.

[0085] The above is merely a schematic description of the target detection model provided by the present application, and the present application is not limited thereto.

[0086] See below Figure 4 , Figure 4 is a flow chart of a target detection method according to an embodiment of the present invention. Figure 4 The following steps are shown:

[0087] Step S211: The central processing unit receives video data captured by multiple video capture devices and converts the received video data into multiple image frames.

[0088] Step S221: the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model, and the target detection model is trained by the target detection model training method as described above.

[0089] Step S212: The central processing unit obtains the target detection result output by the target detection model.

[0090] The target detection system of the present invention respectively executes different steps in target detection model training and target detection through the central processing unit and the image processing unit, and makes the target detection model a convolutional neural network, so that the training process and the detection process of the target detection model are executed in the image processing unit. On the one hand, the computing power of the image processing unit is used to perform complex calculations, which reduces the consumption of computing resources of the central processing unit and further improves the overall target detection efficiency of the system; on the other hand, compared with the frame difference and optical flow algorithms, the target detection accuracy of the convolutional neural network is higher, thereby also improving the accuracy of target detection.

[0091] See below Figure 5 , Figure 5 The flowchart of selecting a central processing unit or an image processing unit to perform target detection according to an embodiment of the present invention. Figure 5 The following steps are shown:

[0092] Step S201: Acquire the current load parameters of the central processing unit.

[0093] Specifically, the load parameters may include, but are not limited to, the utilization rate of the central processing unit, memory, data transmission speed, etc.

[0094] Furthermore, by collecting historical load parameters of the central processing unit, a load parameter range that can indicate that the central processing unit is in an idle state and a load parameter range that can indicate that the central processing unit is in a busy state can be calculated or set.

[0095] Step S202: If the load parameter indicates that the central processing unit is in an idle state, the central processing unit performs target detection on the multiple image frames converted by the central processing unit based on a frame difference algorithm.

[0096] Step S203: If the load parameter indicates that the central processing unit is in a busy state, the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model.

[0097] Therefore, through the above steps, the central processing unit can perform complex calculations when the central processing unit is in an idle state, and the image processing unit can perform complex calculations when the central processing unit is in a busy state, so as to utilize the existing algorithm of the central processing unit without modifying the current algorithm.

[0098] Furthermore, in some variations, when step S202 is executed, it can be further determined whether the image processing unit is in a busy state. If the image processing unit is in a busy state, the central processing unit performs complex calculations. If the image processing unit is in an idle state, the image processing unit can perform complex calculations. In this variation, it is precisely because the accuracy of the detection algorithm of the target detection model executed by the image processing unit is higher than the frame difference algorithm and the border search algorithm of the central processing unit. Therefore, when the image processing unit is in an idle state, the image processing unit can perform complex calculations first. In this embodiment, the image processing unit can perform further target recognition calculations based on the obtained target border. Thus, when the image processing unit is performing target recognition of multiple target frames, which affects the computing power of the image processing unit, the central processing unit can perform target recognition based on the original algorithm.

[0099] In some specific implementations, after testing, the training speed of the target detection model of the present application is 8 to 17fps (Frames Per Second), and the training speed meets the actual project requirements. Among the model indicators, the regional intersection ratio is 0.66, the accuracy is 0.76, and the recall rate is 0.89. The three major indicators are better than the results of the frame difference method. Furthermore, when executing the target detection model training method and target detection method of the present application, the resource overhead of the image processing unit is 1.7GB, and the resource overhead of the central processing unit is 43% (utilization rate), among which the load of the central processing unit is also greatly optimized compared with the existing algorithm.

[0100] The above is merely a schematic description of multiple implementations of the present application. Each embodiment may be implemented individually or in combination, and the present application is not limited thereto.

[0101] like Figure 6 As shown, the embodiment of the present invention further provides a target detection model training system for implementing the target detection model training method, and the system 310 includes:

[0102] The first receiving and converting module 311 is used for receiving the video data collected by the video acquisition device by the central processing unit, and converting the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set;

[0103] The training annotation module 312 is used for executing the pre-trained target detection model by the image processing unit to perform training annotation on multiple image frames in the training set;

[0104] The training module 313 is used for the image processing unit to input the multiple image frames and their training annotations in the training set into the target detection model for training, and the target detection model is a convolutional neural network model;

[0105] The test annotation module 314 is used for executing the pre-trained target detection model by the image processing unit to perform test annotation on multiple image frames in the test set;

[0106] The first input module 315 is used for the image processing unit to input multiple image frames in the test set into the target detection model to obtain a target detection result;

[0107] The test evaluation module 316 is used for the central processing unit to compare the test annotation and the target detection result to obtain the test evaluation result of the target detection model.

[0108] The functions of each module in the target detection model training system of the present invention can be implemented by using the specific implementation methods of the above steps, which will not be repeated here.

[0109] like Figure 7 As shown, the embodiment of the present invention further provides a target detection system for implementing the target detection method, and the system 320 includes:

[0110] The second receiving and converting module 321 is used for receiving the video data collected by multiple video collecting devices by the central processing unit, and converting the received video data into multiple image frames;

[0111] The first input module 322 is used for inputting the plurality of image frames converted by the central processing unit into the target detection model by the image processing unit, and the target detection model is trained by the target detection model training method as described above;

[0112] The output module 323 is used for the central processing unit to obtain the target detection result output by the target detection model.

[0113] The functions of each module in the target detection system of the present invention can be implemented by using the specific implementation methods of the above steps, which will not be described in detail here.

[0114] The target detection model training system and the target detection system of the present invention respectively execute different steps in target detection model training and target detection through a central processing unit and an image processing unit, and make the target detection model a convolutional neural network, so that the training process and the detection process of the target detection model are executed in the image processing unit. On the one hand, the computing power of the image processing unit is used to perform complex calculations, which reduces the consumption of computing resources of the central processing unit and further improves the overall target detection efficiency of the system; on the other hand, compared with the frame difference and optical flow algorithms, the target detection accuracy of the convolutional neural network is higher, thereby also improving the accuracy of target detection.

[0115] Figure 6 and Figure 7 The target detection model training system and the target detection system provided by the present invention are only schematically shown. Without violating the concept of the present invention, the splitting, merging and adding of modules are all within the protection scope of the present invention. The target detection model training system and the target detection system provided by the present invention can be implemented by software, hardware, firmware, plug-ins and any combination thereof, and the present invention is not limited thereto.

[0116] An embodiment of the present invention also provides an electronic device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to execute the steps of the target detection model training method and / or the target detection method by executing the executable instructions.

[0117] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0118] Refer to the following Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0119] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0120] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above target detection model training method and / or target detection method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0121] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0122] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0123] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0124] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] In the electronic device, when the program in the memory is executed by the processor, the steps of the target detection model training method and / or the target detection method are implemented. Therefore, the device can also obtain the technical effects of the above-mentioned target detection model training method and / or the target detection method.

[0126] An embodiment of the present invention also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the target detection model training method and / or the target detection method. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above target detection model training method and / or target detection method section of this specification.

[0127] refer to Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0128] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0130] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0131] When the program in the computer storage medium is executed by the processor, the steps of the target detection model training method and / or the target detection method are implemented. Therefore, the computer storage medium can also obtain the technical effects of the target detection model training method and / or the target detection method.

[0132] The target detection model training method, system, device and storage medium of the present invention have the following beneficial effects:

[0133] The present application uses a central processing unit and an image processing unit to respectively execute different steps in target detection model training and target detection, and makes the target detection model a convolutional neural network, so that the training process and the detection process of the target detection model are executed in the image processing unit. On the one hand, the computing power of the image processing unit is used to perform complex calculations, which reduces the consumption of computing resources of the central processing unit and further improves the overall target detection efficiency of the system; on the other hand, compared with frame difference and optical flow algorithms, the target detection accuracy of the convolutional neural network is higher, thereby also improving the accuracy of target detection.

[0134] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A target detection model training method, characterized in that: The steps include: The central processing unit receives video data collected by the video collection device, and converts the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set; The image processing unit executes a pre-trained target detection model to perform training annotation on a plurality of image frames in the training set; The image processing unit inputs the multiple image frames and their training annotations in the training set into the target detection model for training, and obtains the current load parameter of the central processing unit; if the load parameter indicates that the central processing unit is in an idle state, the central processing unit performs target detection on the multiple image frames converted by the central processing unit based on a frame difference algorithm; if the load parameter indicates that the central processing unit is in a busy state, the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model, and the target detection model is a convolutional neural network model; The image processing unit executes a pre-trained target detection model to perform test annotation on multiple image frames in the test set; The image processing unit inputs a plurality of image frames in the test set into the target detection model to obtain a target detection result; The central processing unit compares the test annotation and the target detection result to obtain a test evaluation result of the target detection model.

2. The target detection model training method according to claim 1, characterized in that: The loss function of the target detection model is calculated based on the KL distance, the background target difference penalty term and the area intersection and union ratio, wherein the KL distance is the discreteness of the probability distribution of the training annotations and the probability distribution of the predicted output of the target detection model, the background target difference penalty term is the reciprocal of the difference between the average brightness of the target area of ​​the predicted output of the target detection model and the background area, and the area intersection and union ratio is the intersection and union ratio of the target area of ​​the predicted output of the target detection model and the target area of ​​the training annotation.

3. The target detection model training method according to claim 2, characterized in that: The loss function L of the target detection model is calculated according to the following formula: L=a*KLD+b*penalization+c*(1-IoU), Among them, a, b, c are weights, and the sum of a, b, c is 1, KLD is the KL distance, penalization is the background target difference penalty term, and IoU is the region intersection over union ratio.

4. The target detection model training method according to claim 1, characterized in that: The central processing unit simultaneously receives video data collected by multiple video acquisition devices, and the target detection model is used to perform target detection on the video data collected by the multiple video acquisition devices.

5. The target detection model training method according to claim 1, characterized in that: The target detection model is trained for multiple iterations, and the pre-trained target detection model is the target detection model trained for the previous iteration.

6. A target detection method, characterized in that: include: The central processing unit receives video data collected by multiple video collection devices and converts the received video data into multiple image frames; The image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model to obtain the current load parameter of the central processing unit; if the load parameter indicates that the central processing unit is in an idle state, the central processing unit performs target detection on the multiple image frames converted by the central processing unit based on a frame difference algorithm; if the load parameter indicates that the central processing unit is in a busy state, the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model, and the target detection model is trained by the target detection model training method according to any one of claims 1 to 5; The central processing unit obtains the target detection result output by the target detection model.

7. A target detection model training system, characterized in that: include: A receiving and converting module, configured to receive video data collected by a video acquisition device by a central processing unit, and convert the received video data into a plurality of image frames, wherein the plurality of image frames are divided into a training set and a test set; A training and annotation module, configured to execute a pre-trained target detection model by an image processing unit to perform training and annotation on a plurality of image frames in the training set; A training module, configured to input the multiple image frames and their training annotations in the training set into a target detection model for training by the image processing unit, and obtain a current load parameter of the central processing unit; if the load parameter indicates that the central processing unit is in an idle state, the central processing unit performs target detection on the multiple image frames converted by the central processing unit based on a frame difference algorithm; if the load parameter indicates that the central processing unit is in a busy state, the image processing unit inputs the multiple image frames converted by the central processing unit into the target detection model, and the target detection model is a convolutional neural network model; A test annotation module, configured to execute a pre-trained target detection model by the image processing unit to perform test annotation on a plurality of image frames in the test set; An input module, configured to input a plurality of image frames in the test set into the target detection model by the image processing unit to obtain a target detection result; A test evaluation module is used for the central processing unit to compare the test annotation and the target detection result to obtain a test evaluation result of the target detection model.

8. An electronic device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to perform, via executing the executable instructions: The steps of the target detection model training method according to any one of claims 1 to 5; and / or The steps of the target detection method described in claim 6.

9. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by the processor, it realizes: The steps of the target detection model training method according to any one of claims 1 to 5; and / or The steps of the target detection method described in claim 6.

Images