Target detection method and device

Abstract

Embodiments of the present application disclose a target detection method and device. The method comprises: inputting training sample images into a neural network in sequence, taking the output of the last layer or multiple layers of a backbone network as a first feature map, and taking the first feature map as the input of a first branch network and a second branch network respectively; when a training termination condition is met, taking a current neural network model as a target detection model; inputting a sample image to be detected into the target detection model, determining the predicted positions of target center points according to a target center point probability prediction mapping diagram output by the first branch network, mapping the predicted positions of the target center points to the sample image to be detected, determining the predicted width and height of a target bounding box corresponding to each target center point according to a target width-height prediction mapping diagram output by the second branch network, and mapping and calibrating the predicted width and height of the target bounding box corresponding to each target center point on the sample image to be detected. Embodiments of the present application improve the speed of target detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target detection method and device. Background Art

[0002] "Smart city" is becoming one of the hottest buzzwords in society today. Cars can drive themselves, self-service supermarkets are more convenient, and roadside parking no longer requires manual scheduling. Artificial intelligence (AI) is making cities smarter.

[0003] Computer vision enables artificial intelligence to "see" and make decisions. Object detection, as one of the most important tasks in computer vision, plays a vital role in various applications.

[0004] Compared with cloud-based AI solutions, edge device deployment not only reduces network communication operating costs, but also offers the advantages of being free from network connectivity restrictions and physical isolation of private data. Edge devices are characterized by low power consumption and low cost.

[0005] Object detection is one of the most important tasks in computer vision, and it is also the basis for further research, such as instance segmentation, pose estimation, tracking, and action recognition. Currently, the most common object detection systems mainly use anchor-based methods. Figure 1 Schematic diagram of existing anchor-based methods for target detection, where the anchor refers to a bounding box with a pre-set fixed w / h (width / height) ratio, such as Figure 1 As shown, a feature map 101 is first extracted from the image to be detected, and then multiple candidate bounding boxes 102 are generated using anchors. Then, the NMS (Non-Maximum Suppression) algorithm is used to merge those highly overlapping boxes 103, and finally the candidate bounding box 104 with the highest confidence is retained as the final detected target.

[0006] In the above method, the NMS algorithm is relatively time-consuming. For the server, this cost is almost negligible. However, for edge devices with limited computing resources, it puts a great pressure.

[0007] In recent years, advances in deep learning have made object detection more accurate and reliable. Besides accuracy, low latency is also a key performance metric for detection systems. Balancing the performance and accuracy of deep learning systems, such as neural networks, is particularly crucial when deployed on edge devices. Summary of the Invention

[0008] In view of this, embodiments of the present invention provide a target detection method and apparatus to improve the speed of target detection.

[0009] The technical solution of the embodiment of the present invention is achieved as follows:

[0010] A target detection method, comprising:

[0011] Input the training sample images into the neural network in sequence, the neural network including: a backbone network and a first branch network and a second branch network, the output of the last layer or layers of the backbone network is used as the first feature map, the first feature map is used as the input of the first branch network and the second branch network respectively, the output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point, the output of the second branch network is: a target width and height prediction map, wherein the two-dimensional value of each point on the map represents: the predicted width and height of the corresponding target bounding box if the point is the target center point; wherein the target center point probability prediction map and the target width and height prediction map are the same size;

[0012] When the training termination condition is met, the current neural network model is used as the target detection model;

[0013] The sample image to be detected is input into the target detection model, and the predicted position of each target center point is determined according to the target center point probability prediction map output by the first branch network. The predicted position of each target center point is mapped to the sample image to be detected. According to the target width and height prediction map output by the second branch network, the predicted width and height of the target bounding box corresponding to each target center point are determined, and the predicted width and height of the target bounding box corresponding to each target center point are mapped and calibrated on the sample image to be detected.

[0014] In the above embodiment, the feature map of the image is extracted through the backbone network of the neural network, and then two tasks are performed simultaneously through two branch networks: detecting the position of the center point of each target and calculating the width and height of the bounding box of each target. Finally, the results of these two tasks are merged to mark each target on the image. The entire process does not require complex calculations, and since the two tasks are performed in parallel, the speed of target detection is greatly improved.

[0015] After the training sample images are sequentially input into the neural network, the method further includes:

[0016] For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map to obtain a second training sample image, calibrate the center point of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to a first preset value, and update the value of each background point to a second preset value to obtain a target center point probability true map of the first training sample image; and calibrate the bounding box of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to: (width of the bounding box corresponding to the target, height of the bounding box corresponding to the target), and update the values ​​of other points to: (third preset value, fourth preset value), to obtain a target width and height true map of the first training sample image;

[0017] Moreover, when the first branch network outputs the target center point probability prediction map of the first training sample image and the second branch network outputs the target width and height prediction map of the first training sample image, the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image is calculated to obtain a first similarity; the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity; the first similarity and the second similarity are weightedly calculated to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network.

[0018] In the above embodiment, by constructing a target center point probability true map and a target width and height true map of the first training sample image, and then calculating the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image, a first similarity is obtained, and the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity. The first similarity and the second similarity are weightedly calculated to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network, so that the neural network can be optimized in the direction of obtaining a more accurate target center point position and a bounding box of the target center point, thereby improving the accuracy of the target detection model.

[0019] The training termination condition is: the training is terminated when the loss function value continues to decrease and stops decreasing.

[0020] The neural network further comprises: a third branch network, and,

[0021] While using the first feature map as input to the first branch network and the second branch network respectively, the method further includes:

[0022] Using the first feature map as input of the third branch network;

[0023] Furthermore, the output of the third branch network is a target center point deviation prediction map, wherein the value of each point on the map represents the deviation between the predicted position and the actual position of the point if the point is the target center point; and the size of the target center point deviation prediction map is the same as the size of the target center point probability prediction map;

[0024] Furthermore, after determining the predicted position of each target center point and before mapping the determined predicted position of each target center point onto the sample image to be detected, the method further includes:

[0025] Determining, based on the target center point deviation prediction map output by the third branch network, a deviation between a predicted position and an actual position of each target center point, and correcting the predicted position of each target center point based on the predicted position of each target center point and the deviation between the predicted position and the actual position of each target center point;

[0026] Furthermore, mapping the predicted positions of the determined target center points onto the sample image to be detected includes:

[0027] The corrected predicted positions of the target center points are mapped onto the sample image to be detected.

[0028] In the above embodiment, the feature map of the image is extracted through the backbone network of the neural network, and then three tasks are performed simultaneously through three branch networks: detecting the position of the center point of each target, calculating the width and height of the bounding box of each target, and calculating the position deviation of the center point of each target. Finally, the output results of the three tasks are merged: first, the output result of the first task is corrected according to the output result of the third task, and then the corrected output result of the first task and the output result of the third task are mapped to the original input image, that is, the detection result of the original input image is obtained, thereby correcting the position deviation of the target center point caused during the detection process, thereby improving the accuracy of target detection.

[0029] After the training sample images are sequentially input into the neural network, the method further includes:

[0030] For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map to obtain a second training sample image, calculate the true position of each target center point in the first training sample image in the second training sample image, and if the true position of any target center point is not an integer, perform a rounding operation on the true position to obtain the actual position of each target center point in the second training sample image; based on the actual position of each target center point in the second training sample image, calibrate each target center point in the second training sample image, and update the value of each calibrated target center point in the second training sample image to: the deviation between the actual position and the true position of the target center point, to obtain a target center point deviation true map of the first training sample image;

[0031] Furthermore, after obtaining the second similarity and before performing weighted calculation on the first similarity and the second similarity, the method further includes:

[0032] Calculating the similarity between the true target center point deviation map of the first training sample image and the predicted target center point deviation map of the first training sample image output by the third branch network to obtain a third similarity;

[0033] Furthermore, performing weighted calculation on the first similarity and the second similarity includes:

[0034] A weighted calculation is performed on the first similarity, the second similarity, and the third similarity, and the weighted calculation result is used as the loss function value.

[0035] In the above embodiment, a target center point probability true map, a target width and height true map, and a target center point deviation prediction map of the first training sample image are constructed, and then the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image is calculated to obtain a first similarity, the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity, and the similarity between the target center point deviation prediction map of the first training sample image and the target center point deviation true map of the first training sample image is calculated to obtain a third similarity, and a weighted calculation is performed on the first similarity, the second similarity, and the third similarity to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network, so that the neural network can be optimized toward obtaining a more accurate target center point position, thereby further improving the accuracy of the target detection model.

[0036] After using the output of the last layer or layers of the backbone network as the first feature map and before using the first feature map as the input of the first branch network and the second branch network, the method further includes:

[0037] The first feature map is upsampled according to a preset upsampling rate, and the second feature map obtained after upsampling is used as the input of the first branch network and the second branch network respectively.

[0038] The above embodiments improve the accuracy of the target detection model.

[0039] Before sequentially inputting the training sample images into the neural network, the method further includes:

[0040] The backbone network is constructed according to the following principle: the backbone network is divided into multiple blocks, wherein at least one block satisfies the following conditions: the block is composed of at least two convolutional layers, and the block satisfies: the dimension of the input data of the block = the dimension of the output data of the block > the number of filters contained in each layer of the block except the last layer.

[0041] The above embodiment reduces the complexity of target detection and further improves the target detection speed while ensuring the accuracy of the target detection model.

[0042] The target is a vehicle.

[0043] A target detection device, comprising:

[0044] A detection model establishment module is used to input training sample images into a neural network in sequence. The neural network includes: a backbone network and a first branch network and a second branch network. The output of the last layer or layers of the backbone network is used as a first feature map. The first feature map is used as the input of the first branch network and the second branch network respectively. The output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point. The output of the second branch network is: a target width and height prediction map, wherein the two-dimensional value of each point on the map represents: the predicted width and height of the corresponding target bounding box if the point is the target center point; wherein the target center point probability prediction map and the target width and height prediction map have the same size; when the training termination condition is met, the current neural network model is used as the target detection model;

[0045] The detection module is used to input the sample image to be detected into the target detection model, determine the predicted position of each target center point according to the target center point probability prediction map output by the first branch network, map the determined predicted position of each target center point to the sample image to be detected, determine the predicted width and height of the target bounding box corresponding to each target center point according to the target width and height prediction map output by the second branch network, map the predicted width and height of the target bounding box corresponding to each target center point and calibrate it on the sample image to be detected.

[0046] A target detection device comprises: a memory and a processor capable of accessing the memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the processor executes the steps of any one of the above methods.

[0047] In an embodiment of the present invention, the feature map of the image is extracted through the backbone network of the neural network, and then two tasks are performed simultaneously through two branch networks: detecting the position of the center point of each target and calculating the width and height of the bounding box of each target. Finally, the results of these two tasks are merged to mark each target on the image. The entire process does not require complex calculations, and since the two tasks are performed in parallel, the speed of target detection is greatly improved.

[0048] A computer-readable medium stores computer-readable instructions, which, when executed by a processor, cause the processor to perform the steps of any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, so that those skilled in the art will understand the above and other features and advantages of the present invention more clearly. In the accompanying drawings:

[0050] Figure 1 Schematic diagram of existing anchor-based methods for target detection;

[0051] Figure 2 A flow chart of a target detection method provided by one embodiment of the present invention;

[0052] Figure 3 A flow chart of a target detection method provided by another embodiment of the present invention;

[0053] Figure 4 A flow chart of a target detection method provided by another embodiment of the present invention;

[0054] Figure 5 A flow chart of a target detection method provided by another embodiment of the present invention;

[0055] Figure 6A schematic diagram of the process of performing target detection using a target detection model provided in an embodiment of the present invention;

[0056] Figure 7 A schematic diagram of inputting a second feature map into three branch networks for calculation and ultimately outputting a target detection result, provided by an embodiment of the present invention;

[0057] Figure 8 An example structural diagram of a funnel block in a neural network provided by the present invention;

[0058] Figure 9 A schematic structural diagram of a target detection device provided by one embodiment of the present invention;

[0059] Figure 10 A schematic structural diagram of a target detection device provided in another embodiment of the present invention.

[0060] The accompanying drawings are numerals as follows:

[0061]

[0062] DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail with reference to the following examples.

[0064] Figure 2 The flowchart of the target detection method provided in one embodiment of the present invention includes the following specific steps:

[0065] Step 201: The training sample images are sequentially input into a neural network, which includes: a backbone network and a first branch network and a second branch network. The output of the last layer or layers of the backbone network is used as a first feature map, and the first feature map is used as the input of the first and second branch networks respectively. The output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point. The output of the second branch network is: a target width and height prediction map, wherein the two-dimensional values ​​of each point on the map represent: the predicted width and height of the corresponding target bounding box if the point is the target center point. The target center point probability prediction map and the target width and height prediction map are the same size.

[0066] Among them, some or all of all training sample images input into the neural network are images containing the target.

[0067] The same size of the target center point probability prediction map and the target width and height prediction map means that the two contain the same number of points, where each point represents one or more points in the training sample image. For example, when the size of the training sample image is a*b, if the sizes of the target center point probability prediction map and the target width and height prediction map are (a / m)*(b / m), then each point in the target center point probability prediction map and the target width and height prediction map respectively represents m*m points at the corresponding position in the training sample image, where a, b, and m are positive integers.

[0068] When the output of the last multiple layers of the backbone network is used as the first feature map, the data of the last multiple layers of output need to be fused. The fusion method can be pre-set, and the embodiment of the present invention does not limit this. For example, a more common splicing method can be used for fusion.

[0069] Step 202: When the training termination condition is met, the current neural network model is used as the target detection model.

[0070] Step 203: Input the sample image to be detected into the target detection model, determine the predicted position of each target center point based on the target center point probability prediction map output by the first branch network, and map the predicted position of each target center point to the sample image to be detected; determine the predicted width and height of the target bounding box corresponding to each target center point based on the target width and height prediction map output by the second branch network, and map and calibrate the predicted width and height of the target bounding box corresponding to each target center point to the sample image to be detected.

[0071] According to the target center point probability prediction map output by the first branch network, the predicted position of each target center point can be determined as follows:

[0072] Set a sliding window of size c*d, where c is the width of the sliding window and d is the height of the sliding window. The sliding step length of the sliding window in the width direction is c, and the sliding step length in the height direction is d. The sliding window slides on the target center point probability prediction map. Every time it slides to a new position, the maximum target center point probability is found among the target center point probabilities of the c*d points in the sliding window. Then, it is determined whether the maximum target center point probability is greater than the preset threshold. If it is greater, the point where the maximum target center point probability is located is considered to be the predicted position of the target center point.

[0073] In the above embodiment, the feature map of the image is extracted through the backbone network of the neural network, and then two tasks are performed simultaneously through two branch networks: detecting the position of the center point of each target and calculating the width and height of the bounding box of each target. Finally, the results of these two tasks are merged to mark each target on the image. The entire process does not require complex calculations, and since the two tasks are performed in parallel, the speed of target detection is greatly improved.

[0074] In the process of neural network training, the setting of loss function is very important. The loss function is used to guide the neural network to correct the optimal model.

[0075] Figure 3 A flow chart of a target detection method provided in another embodiment of the present invention, wherein the specific steps are as follows:

[0076] Step 301: The training sample images are sequentially input into a neural network, which includes: a backbone network and a first branch network and a second branch network. The output of the last layer or layers of the backbone network is used as a first feature map, and the first feature map is used as the input of the first and second branch networks respectively. The output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point. The output of the second branch network is: a target width and height prediction map, wherein the two-dimensional values ​​of each point on the map represent: the predicted width and height of the corresponding target bounding box if the point is the target center point. The target center point probability prediction map and the target width and height prediction map are the same size.

[0077] Step 302: For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map to obtain the second training sample image, calibrate the center point of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to the first preset value, and update the value of each background point to the second preset value to obtain the target center point probability true map of the first training sample image; and, calibrate the bounding box of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to: (width of the bounding box corresponding to the target, height of the bounding box corresponding to the target), and update the values ​​of other points to: (third preset value, fourth preset value), to obtain the target width and height true map of the first training sample image.

[0078] The first preset value and the second preset value may be different; the third preset value and the fourth preset value may be the same or different. In practical applications, the first preset value may be 1, the second preset value may be 0, the third preset value may be 0, and the fourth preset value may be 0.

[0079] Step 303: When the first branch network outputs a target center point probability prediction map of the first training sample image and the second branch network outputs a target width and height prediction map of the first training sample image, the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image is calculated to obtain a first similarity; the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity; a weighted calculation is performed on the first similarity and the second similarity to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network.

[0080] Step 304: When the training termination condition is met, the current neural network model is used as the target detection model.

[0081] The training termination condition can be: the training is terminated when the loss function value continues to decrease and stops decreasing.

[0082] Step 305: Input the sample image to be detected into the target detection model, determine the predicted position of each target center point based on the target center point probability prediction map output by the first branch network, and map the predicted position of each target center point to the sample image to be detected; determine the predicted width and height of the target bounding box corresponding to each target center point based on the target width and height prediction map output by the second branch network, and map and calibrate the predicted width and height of the target bounding box corresponding to each target center point to the sample image to be detected.

[0083] In the above embodiment, by constructing a target center point probability true map and a target width and height true map of the first training sample image, and then calculating the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image, a first similarity is obtained, and the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity. The first similarity and the second similarity are weightedly calculated to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network, so that the neural network can be optimized in the direction of obtaining a more accurate target center point position and a bounding box of the target center point, thereby improving the accuracy of the target detection model.

[0084] In the above embodiment, when converting the first training sample image to the second training sample image, if the width and height coordinates of the target center point are not integers, they need to be converted to integers. During this conversion process, the position of the target center point will deviate. To eliminate this deviation, the following solution is provided:

[0085] Figure 4A flow chart of a target detection method provided in another embodiment of the present invention, wherein the specific steps are as follows:

[0086] Step 401: Input the training sample images into the neural network in sequence. The neural network includes: a backbone network and a first branch network, a second branch network, and a third branch network. The output of the last layer or layers of the backbone network is used as the first feature map, and the first feature map is used as the input of the first, second, and third branch networks respectively. The output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point. The output of the second branch network is: a target width and height prediction map, wherein the two-dimensional values ​​of each point on the map represent: the predicted width and height of the corresponding target bounding box if the point is the target center point. The output of the third branch network is: a target center point deviation prediction map, wherein the value of each point on the map represents: the deviation between the predicted position of the point and the actual position if the point is the target center point. The target center point probability prediction map, the target width and height prediction map, and the target center point deviation prediction map are of the same size.

[0087] Among them, each point in the target center point deviation prediction map represents one or more points in the training sample image. For example: when the size of the training sample image is a*b, if the size of the target center point deviation prediction map is (a / m)*(b / m), then each point in the target center point deviation prediction map represents m*m points at the corresponding position in the training sample image, where m is a positive integer.

[0088] Step 402: When the training termination condition is met, the current neural network model is used as the target detection model.

[0089] Step 403: Input the sample image to be detected into the target detection model, determine the predicted position of each target center point according to the target center point probability prediction map output by the first branch network, determine the deviation between the predicted position and the actual position of each target center point according to the target center point deviation prediction map output by the third branch network, and correct the predicted position of each target center point according to the predicted position of each target center point and the deviation between the predicted position and the actual position of each target center point, and map the corrected predicted position of each target center point to the sample image to be detected, determine the predicted width and height of the target bounding box corresponding to each target center point according to the target width and height prediction map output by the second branch network, and map and calibrate the predicted width and height of the target bounding box corresponding to each target center point on the sample image to be detected.

[0090] In the above embodiment, the feature map of the image is extracted through the backbone network of the neural network, and then three tasks are performed simultaneously through three branch networks: detecting the position of the center point of each target, calculating the width and height of the bounding box of each target, and calculating the position deviation of the center point of each target. Finally, the output results of the three tasks are merged: first, the output result of the first task is corrected according to the output result of the third task, and then the corrected output result of the first task and the output result of the third task are mapped to the original input image, that is, the detection result of the original input image is obtained, thereby correcting the position deviation of the target center point caused during the detection process, thereby improving the accuracy of target detection.

[0091] Figure 5 A flow chart of a target detection method provided in another embodiment of the present invention, wherein the specific steps are as follows:

[0092] Step 501: Input the training sample images into the neural network in sequence. The neural network includes: a backbone network and a first branch network, a second branch network, and a third branch network. The output of the last layer or layers of the backbone network is used as the first feature map, and the first feature map is used as the input of the first, second, and third branch networks respectively. The output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point. The output of the second branch network is: a target width and height prediction map, wherein the two-dimensional value of each point on the map represents: the predicted width and height of the corresponding target bounding box if the point is the target center point. The output of the third branch network is: a target center point deviation prediction map, wherein the value of each point on the map represents: the deviation between the predicted position of the point and the actual position if the point is the target center point. The target center point probability prediction map, the target width and height prediction map, and the target center point deviation prediction map are of the same size.

[0093] Among them, each point in the target center point deviation prediction map represents one or more points in the training sample image. For example: when the size of the training sample image is a*b, if the size of the target center point deviation prediction map is (a / m)*(b / m), then each point in the target center point deviation prediction map represents m*m points at the corresponding position in the training sample image, where m is a positive integer.

[0094] Step 502: For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map, and obtain the second training sample image. Calibrate the center point of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to the first preset value, and update the value of each background point to the second preset value to obtain the target center point probability true map of the first training sample image; and calibrate the bounding box of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to: (width of the bounding box corresponding to the target, height of the bounding box corresponding to the target), and update the values ​​of other points to: The value is updated to: (third preset value, fourth preset value), and a target width and height true mapping map of the first training sample image is obtained; and the true position of each target center point in the first training sample image is calculated in the second training sample image, and, if the true position of any target center point is not an integer, the true position is rounded to obtain the actual position of each target center point in the second training sample image, and according to the actual position of each target center point in the second training sample image, each target center point is calibrated in the second training sample image, and the value of each calibrated target center point in the second training sample image is updated to: the deviation between the actual position of the target center point and the true position, to obtain a target center point deviation true mapping map of the first training sample image.

[0095] Step 503: When the first branch network outputs a target center point probability prediction map of the first training sample image, the second branch network outputs a target width and height prediction map of the first training sample image, and the third branch network outputs a target center point deviation prediction map of the first training sample image, the similarity between the target center point probability prediction map of the first training sample image and the target center point true probability map of the first training sample image is calculated to obtain a first similarity; the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity; the similarity between the target center point deviation prediction map of the first training sample image and the target center point deviation true map of the first training sample image is calculated to obtain a third similarity; the first similarity, the second similarity, and the third similarity are weightedly calculated to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network.

[0096] Step 504: When the training termination condition is met, the current neural network model is used as the target detection model.

[0097] Step 505: Input the sample image to be detected into the target detection model, determine the predicted position of each target center point according to the target center point probability prediction map output by the first branch network, determine the deviation between the predicted position and the actual position of each target center point according to the target center point deviation prediction map output by the third branch network, and correct the predicted position of each target center point according to the predicted position of each target center point and the deviation between the predicted position and the actual position of each target center point, and map the corrected predicted position of each target center point to the sample image to be detected, determine the predicted width and height of the target bounding box corresponding to each target center point according to the target width and height prediction map output by the second branch network, and map and calibrate the predicted width and height of the target bounding box corresponding to each target center point on the sample image to be detected.

[0098] In the above embodiment, a target center point probability true map, a target width and height true map, and a target center point deviation prediction map of the first training sample image are constructed, and then the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image is calculated to obtain a first similarity, the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity, and the similarity between the target center point deviation prediction map of the first training sample image and the target center point deviation true map of the first training sample image is calculated to obtain a third similarity, and a weighted calculation is performed on the first similarity, the second similarity, and the third similarity to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network, so that the neural network can be optimized toward obtaining a more accurate target center point position, thereby further improving the accuracy of the target detection model.

[0099] Figure 6 A schematic diagram of the process of performing target detection using a target detection model provided by an embodiment of the present invention is given, wherein 601 is an image to be detected, and the image to be detected 601 is input into the backbone network 602 of the target detection model for feature extraction, 603 is the features output by the last two layers of the backbone network, 604 is splicing the features output by the last two layers of the backbone network to obtain a first feature map, 605 is upsampling the first feature map to obtain a second feature map, and the second feature map is simultaneously input into three branch networks 606, 607, and 608 for calculation respectively, 609 is a target center point probability prediction map output by the branch network 606, 610 is a target width and height prediction map output by the branch network 607, and 611 is a target center point deviation prediction map output by the branch network 608. According to the outputs of the three branch networks, a target detection result 612 can be obtained.

[0100] Figure 7A schematic diagram of an embodiment of the present invention providing an input of a second feature map into three branch networks for calculation and final output of a target detection result, wherein 71 is the second feature map, 72 is the target center point probability prediction map output by the first branch network, 73 is the target width and height prediction map output by the second branch network, 74 is the target center point deviation prediction map output by the third branch network, and 75 is the target calibrated according to 72, 73 and 74.

[0101] In an optional embodiment, in order to improve the accuracy of the target detection model, before the first feature map is used as the input of the first, second and third branch networks respectively, the first feature map is further upsampled according to a preset upsampling rate, and the second feature map obtained after upsampling is used as the input of the first, second and third branch networks respectively.

[0102] In an optional embodiment, in order to reduce the complexity of target detection and further improve the target detection speed while ensuring the accuracy of the target detection model, the backbone network can be constructed according to the following principles: the backbone network is divided into multiple blocks, wherein at least one block satisfies the following conditions: the block is composed of at least two convolutional layers, and the block satisfies: the dimension of the input data of the block = the dimension of the output data of the block > the number of filters contained in each layer of the block except the last layer.

[0103] In practical applications, blocks that meet the above conditions are called funnel blocks. Figure 8 This is an example structural diagram of the funnel block in the neural network provided by the present invention. Figure 8 As shown, the block contains 3 convolutional layers, where the dimension of the input data is 256, the first layer 81 contains 64 filters, and the convolution kernel size used is 1*1, the second layer 82 contains 64 filters, and the convolution kernel size used is 3*3, and the last layer 83 contains 256 filters, and the convolution kernel size used is 1*1. The output data of the last layer is added to the input data to obtain the output data of the block.

[0104] Figure 9 This is a schematic diagram of the structure of a target detection device 90 provided in one embodiment of the present invention. The device mainly includes:

[0105] A detection model establishment module 91 is used to sequentially input training sample images into a neural network, the neural network comprising: a backbone network and a first branch network and a second branch network, the output of the last layer or layers of the backbone network being used as a first feature map, the first feature map being used as input to the first and second branch networks respectively, the output of the first branch network being: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point, the output of the second branch network being: a target width and height prediction map, wherein the two-dimensional values ​​of each point on the map represent: the predicted width and height of the corresponding target bounding box if the point is the target center point; wherein the target center point probability prediction map and the target width and height prediction map are of the same size; when the training termination condition is met, the current neural network model is used as the target detection model;

[0106] The detection module 92 is used to input the sample image to be detected into the target detection model trained by the detection model establishment module 91, determine the predicted position of each target center point according to the target center point probability prediction map output by the first branch network, map the predicted position of each target center point to the sample image to be detected, determine the predicted width and height of the target bounding box corresponding to each target center point according to the target width and height prediction map output by the second branch network, map the predicted width and height of the target bounding box corresponding to each target center point and calibrate it on the sample image to be detected.

[0107] Figure 10 A structural diagram of a target detection device 10 provided for another embodiment of the present invention, the device comprising: a memory 11 and a processor 12 that can access the memory 11, the memory 11 stores instructions, and when the instructions are executed by the processor 12, the processor 12 executes the steps of the method described in steps 201-203, or steps 301-305, or steps 401-403, or steps 501-505.

[0108] The embodiments of the present invention can be applied to vehicle detection scenarios.

[0109] In addition, the target detection model in the embodiment of the present invention can be further loaded into the tensorRT quantization model for optimization, so that the parameters can be further compressed without significantly reducing the accuracy, thereby avoiding memory consumption and speeding up the calculation speed, so that real-time operation on the edge device can be achieved.

[0110] Figure 9This is a structural diagram of a positioning device 90 provided in one embodiment of the present invention. The device mainly includes: a memory 91 and a processor 92 that can access the memory 91. The memory 91 stores instructions. When the instructions are executed by the processor 92, the processor 92 executes the steps of the method described in steps 201-204, or steps 301-307, or steps 401-404, or steps 601-608, or steps 701-705, or steps 801-810.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target detection method, characterized in that: include: Input the training sample images into the neural network in sequence, the neural network including: a backbone network and a first branch network and a second branch network, the output of the last layer or layers of the backbone network is used as the first feature map, the first feature map is used as the input of the first branch network and the second branch network respectively, the output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point, the output of the second branch network is: a target width and height prediction map, wherein the two-dimensional value of each point on the map represents: the predicted width and height of the corresponding target bounding box if the point is the target center point; wherein the target center point probability prediction map and the target width and height prediction map are the same size; Before sequentially inputting the training sample images into the neural network, the method further includes: The backbone network is constructed according to the following principle: the backbone network is divided into multiple blocks, wherein at least one block satisfies the following conditions: the block is composed of at least two convolutional layers, and the block satisfies: the dimension of the input data of the block = the dimension of the output data of the block > the number of filters contained in each layer of the block except the last layer; When the training termination condition is met, the current neural network model is used as the target detection model; The sample image to be detected is input into the target detection model, and the predicted position of each target center point is determined according to the target center point probability prediction map output by the first branch network. The predicted position of each target center point is mapped to the sample image to be detected. According to the target width and height prediction map output by the second branch network, the predicted width and height of the target bounding box corresponding to each target center point are determined, and the predicted width and height of the target bounding box corresponding to each target center point are mapped and calibrated on the sample image to be detected.

2. The method according to claim 1, characterized in that After the training sample images are sequentially input into the neural network, the method further includes: For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map to obtain a second training sample image, calibrate the center point of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to a first preset value, and update the value of each background point to a second preset value to obtain a target center point probability true map of the first training sample image; and calibrate the bounding box of each target in the second training sample image, and update the value of the center point of each target in the second training sample image to: (width of the bounding box corresponding to the target, height of the bounding box corresponding to the target), and update the values ​​of other points to: (third preset value, fourth preset value), to obtain a target width and height true map of the first training sample image; wherein the first preset value and the second preset value are different; Moreover, when the first branch network outputs the target center point probability prediction map of the first training sample image and the second branch network outputs the target width and height prediction map of the first training sample image, the similarity between the target center point probability prediction map of the first training sample image and the target center point probability true map of the first training sample image is calculated to obtain a first similarity; the similarity between the target width and height prediction map of the first training sample image and the target width and height true map of the first training sample image is calculated to obtain a second similarity; the first similarity and the second similarity are weightedly calculated to obtain a loss function value of the entire neural network, and the loss function value is back-propagated along the neural network.

3. The method according to claim 2, characterized in that The training termination condition is: the training is terminated when the loss function value continues to decrease and stops decreasing.

4. The method according to claim 1 or 2, characterized in that The neural network further comprises: a third branch network, and, While using the first feature map as input to the first branch network and the second branch network respectively, the method further includes: Using the first feature map as input of the third branch network; Furthermore, the output of the third branch network is a target center point deviation prediction map, wherein the value of each point on the map represents the deviation between the predicted position and the actual position of the point if the point is the target center point; and the size of the target center point deviation prediction map is the same as the size of the target center point probability prediction map; Furthermore, after determining the predicted position of each target center point and before mapping the determined predicted position of each target center point onto the sample image to be detected, the method further includes: Determining, based on the target center point deviation prediction map output by the third branch network, a deviation between a predicted position and an actual position of each target center point, and correcting the predicted position of each target center point based on the predicted position of each target center point and the deviation between the predicted position and the actual position of each target center point; Furthermore, mapping the predicted positions of the determined target center points onto the sample image to be detected includes: The corrected predicted positions of the target center points are mapped onto the sample image to be detected.

5. The method according to claim 4, characterized in that After the training sample images are sequentially input into the neural network, the method further includes: For the currently input training sample image, set it as the first training sample image, convert the size of the first training sample image to the same size as the target center point probability prediction map to obtain a second training sample image, calculate the true position of each target center point in the first training sample image in the second training sample image, and if the true position of any target center point is not an integer, perform a rounding operation on the true position to obtain the actual position of each target center point in the second training sample image; based on the actual position of each target center point in the second training sample image, calibrate each target center point in the second training sample image, and update the value of each calibrated target center point in the second training sample image to: the deviation between the actual position and the true position of the target center point, to obtain a target center point deviation true map of the first training sample image; Furthermore, after obtaining the second similarity and before performing weighted calculation on the first similarity and the second similarity, the method further includes: Calculating the similarity between the true target center point deviation map of the first training sample image and the predicted target center point deviation map of the first training sample image output by the third branch network to obtain a third similarity; Furthermore, performing weighted calculation on the first similarity and the second similarity includes: A weighted calculation is performed on the first similarity, the second similarity, and the third similarity, and the weighted calculation result is used as the loss function value.

6. The method according to claim 1, characterized in that After using the output of the last layer or layers of the backbone network as the first feature map and before using the first feature map as the input of the first branch network and the second branch network, the method further includes: The first feature map is upsampled according to a preset upsampling rate, and the second feature map obtained after upsampling is used as the input of the first branch network and the second branch network respectively.

7. The method according to claim 1, characterized in that The target is a vehicle.

8. A target detection device (90), characterized in that include: A detection model establishment module (91) is used to sequentially input training sample images into a neural network, wherein the neural network includes: a backbone network and a first branch network and a second branch network; the output of the last layer or layers of the backbone network is used as a first feature map; the first feature map is used as input to the first branch network and the second branch network respectively; the output of the first branch network is: a target center point probability prediction map, wherein the value of each point on the map represents the probability of the point being the target center point; the output of the second branch network is: a target width and height prediction map, wherein the two-dimensional value of each point on the map represents: the predicted width and height of the corresponding target bounding box if the point is the target center point; wherein the target center point probability prediction map and the target width and height prediction map have the same size; when the training termination condition is met, the current neural network model is used as the target detection model; Before sequentially inputting the training sample images into the neural network, the method further includes: The backbone network is constructed according to the following principle: the backbone network is divided into multiple blocks, wherein at least one block satisfies the following conditions: the block is composed of at least two convolutional layers, and the block satisfies: the dimension of the input data of the block = the dimension of the output data of the block > the number of filters contained in each layer of the block except the last layer; A detection module (92) is used to input the sample image to be detected into the target detection model, determine the predicted position of each target center point based on the target center point probability prediction map output by the first branch network, map the determined predicted position of each target center point onto the sample image to be detected, determine the predicted width and height of the target bounding box corresponding to each target center point based on the target width and height prediction map output by the second branch network, map the predicted width and height of the target bounding box corresponding to each target center point and calibrate them onto the sample image to be detected.

9. A target detection device (10), characterized in that: include: A memory (11) and a processor (12) accessible to the memory (11), the memory (11) storing instructions which, when executed by the processor (12), cause the processor (12) to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer-readable medium stores computer-readable instructions, which, when executed by a processor, enable the processor to perform the method according to any one of claims 1 to 7.

Images