Target detection method, electronic device, medium and vehicle

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically provides a target detection method, an electronic device, a medium and a vehicle, aiming to solve the technical problem of low detection accuracy corresponding to the target detection method in the field of existing vehicle automatic driving. To this end, the target detection method of the present invention includes: obtaining an image to be detected containing a target detection object; inputting the image to be detected into a student network trained by a knowledge distillation method to obtain a target detection object, and training the student network using the knowledge distillation method includes: inputting a traffic scene training image into a trained teacher network to obtain a first feature map and a dynamic area; inputting a traffic scene training image into a student network to be trained to obtain a second feature map and a detection loss part; determining the distillation loss part based on the first feature map, the second feature map and the dynamic area; and training the student network based on the distillation loss part and the detection loss part. In this way, the target detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically provides a target detection method, electronic equipment, a medium and a vehicle. Background Art

[0002] At present, in the field of vehicle autonomous driving, the detection of targets such as pedestrians, traffic lights, obstacles, and lane lines is implemented based on network models. Existing network models usually use efficient lightweight models. When using knowledge distillation to train lightweight models, the constraints between the lightweight model and the teacher network are small. Faced with a large amount of training data in an underfitting state, the lightweight model's expressive ability is difficult to learn the effective features in the teacher network, resulting in poor results of existing distillation methods, which in turn leads to low target detection accuracy in autonomous driving detection, which is difficult to meet actual needs.

[0003] Accordingly, the art needs a new target detection solution to solve the above problems. Summary of the invention

[0004] In order to overcome the above defects, the present invention is proposed to solve or at least partially solve the technical problem of low detection accuracy of target detection methods in the field of existing vehicle automatic driving. The present invention provides a target detection method, electronic equipment, medium and vehicle.

[0005] In a first aspect, the present invention provides a target detection method, comprising the following steps: obtaining an image to be detected containing a target detection object; inputting the image to be detected into a student network trained using a knowledge distillation method to obtain a target detection object; wherein, training the student network using the knowledge distillation method comprises: inputting a traffic scene training image into a trained teacher network to obtain a first feature map and a dynamic area, wherein the teacher network comprises a first feature pyramid network and a region candidate network, the first feature pyramid network outputs the first feature map, and the region candidate network inputs the first feature map and outputs the dynamic area; inputting the traffic scene training image into a student network to be trained to obtain a second feature map and a detection loss part, wherein the student network comprises a second feature pyramid network, the second feature pyramid network outputs the second feature map, and the student network obtains the detection loss part based on a true label; determining a distillation loss part based on the first feature map, the second feature map and the dynamic area; and training the student network based on the distillation loss part and the detection loss part.

[0006] In one embodiment, the region candidate network inputs the first feature map and outputs the dynamic region, including: normalizing the output V(F) of the region candidate network to obtain a normalized feature map:

[0007] Map = σ(V(F));

[0008] The normalized feature map is thresholded to generate the dynamic region:

[0009]

[0010] Among them, σ(·) represents the normalization function, WeightedMap x,y It represents the weight of the dynamic area at the coordinate (x, y), and threshold represents the area threshold.

[0011] In one embodiment, determining the distillation loss part based on the first feature map, the second feature map and the dynamic area includes: using the dynamic area to weight the distillation loss function, wherein the distillation loss function is used to constrain the first feature map and the second feature map to remain consistent.

[0012] In one embodiment, the step of weighting the distillation loss function using the dynamic region includes:

[0013]

[0014] Among them, LOSS represents the weighted square loss function, F t,x,y Indicates the feature of the first feature map at the coordinate (x, y), F s,x,y Indicates the feature of the second feature map at the coordinate (x, y), WeightedMap x,y represents the weight of the dynamic area at the coordinate (x, y), and N is the product of the length and width of the second feature map.

[0015] In one embodiment, training the student network based on the distillation loss part and the detection loss part includes: taking the sum of the distillation loss part and the detection loss part as the global loss of the student network; adjusting the network parameters of the student network based on the global loss until the global loss meets a preset condition.

[0016] In one embodiment, before determining the distillation loss part based on the first feature map, the second feature map and the dynamic area, the method further includes: inputting the second feature map into a FitNet network so that the second feature map is aligned with the first feature map.

[0017] In one embodiment, the region candidate network is any one of RPN, R-CNN and faster-R-CNN.

[0018] In a second aspect, an electronic device is provided, which includes a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, and the program codes are suitable for being loaded and run by the processor to execute any of the target detection methods described above.

[0019] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored in the computer-readable storage medium, wherein the program codes are suitable for being loaded and run by a processor to execute any of the target detection methods described above.

[0020] In a fourth aspect, a vehicle is provided, comprising the electronic device described above.

[0021] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0022] The present invention provides a target detection method, which mainly inputs an acquired image to be detected into a trained student network to obtain a target detection object. In the process of training the student network, a distillation loss part is determined by obtaining a dynamic area, a first feature map from a teacher network, and a second feature map obtained from the student network. Then, the student network is distilled using the distillation loss part and the detection loss part corresponding to the student network. The dynamic area with confidence is used to guide the weights in the distillation, which effectively avoids the student network from fitting background noise. At the same time, the learning of the features of the detected object is strengthened, and the distillation effect is improved. The trained student network has higher accuracy. When it is used for target detection in the field of autonomous driving, it is beneficial to improve the detection accuracy of the detected target object, thereby achieving the effect of safe driving.

[0023] By normalizing the output of the region candidate network, a normalized feature map is obtained, and the normalized feature map is thresholded to generate a dynamic region, thereby distinguishing the foreground from the background, providing technical support for the subsequent determination of the distillation loss function. In addition, the normalized processing can further improve the distillation effect and improve the training accuracy of the student network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0025] Figure 1 is a schematic flow chart of main steps of a target detection method according to an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of the overall process of knowledge distillation according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of the structure of an RPN network according to an embodiment of the present invention;

[0028] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0030] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0031] At present, the detection of targets such as pedestrians, traffic lights, obstacles and lane lines is implemented based on network models. Existing network models usually use efficient lightweight models. When using knowledge distillation to train lightweight models, the constraints between the lightweight model and the teacher network are small. Faced with a large amount of training data in an underfitting state, the lightweight model's expression ability is difficult to learn the effective features in the teacher network, resulting in poor model distillation effect, which in turn leads to low target detection accuracy in autonomous driving detection. To this end, the present application proposes a target detection method and a storage medium, which mainly inputs the acquired image to be detected into a trained student network to obtain a target detection object. In the process of training the student network, the distillation loss part is determined by obtaining the dynamic area and the first feature map from the teacher network and the second feature map obtained by the student network, and then the student network is trained using the distillation loss part and the detection loss part corresponding to the student network. In this way, the learning ability of the student network to the teacher network is increased, the distillation effect is improved, and the trained student network has higher accuracy. When it is used for target detection in the field of autonomous driving, it is beneficial to improve the detection accuracy of the detected target object, thereby achieving the effect of safe driving.

[0032] See attached Figure 1 and 2 The main steps of the knowledge distillation method according to one embodiment of the present invention are described in detail below. Figure 1 As shown, the knowledge distillation method in the embodiment of the present invention mainly includes the following steps S11-S12.

[0033] Step S11: Acquire an image to be detected that contains a target detection object.

[0034] In a specific implementation, the target detection object may be an obstacle, a pedestrian, a traffic light, a lane line, etc., but is not limited thereto.

[0035] The hardware used to obtain the image to be detected may include a vehicle-mounted camera, Lidar, Radar, etc.

[0036] Step S12: Input the image to be detected into the student network trained by the knowledge distillation method to obtain the target detection object, wherein the training of the student network by the knowledge distillation method can be achieved through the following steps S101 to S104.

[0037] Step S101: Input a traffic scene training image into a trained teacher network to obtain a first feature map and a dynamic region, wherein the teacher network includes a first feature pyramid network and a region candidate network, the first feature pyramid network outputs a first feature map, and the region candidate network inputs the first feature map and outputs a dynamic region.

[0038] Generally speaking, traffic scene training images refer to traffic scene image data used to train network models, which contain manually marked labels such as object categories and locations.

[0039] In the scenario of autonomous driving, since the target detection model as a student network can be used to detect target objects such as obstacles, pedestrians, traffic lights and lane lines, the traffic scene training images in this application can be collected training images containing target objects such as obstacles, pedestrians, traffic lights and lane lines.

[0040] Specifically, Figure 2 As shown, the teacher network in the present application includes a backbone network, a first feature pyramid network and a region candidate network, wherein the first feature map output by the first feature pyramid network passes through the region candidate network to obtain a binary classification result corresponding to each feature map, that is, the background and foreground. The present application uses the foreground in the binary classification result as a dynamic area. When the student network is used for target detection, the dynamic area is the area that reflects the location of the target to be detected.

[0041] In a specific implementation, the region candidate network in the present application may be any one of RPN, R-CNN and faster-R-CNN.

[0042] Take RPN as an example. Figure 3 As shown in the figure, it includes a convolution layer, a classifier and a regressor. The feature map output by the feature pyramid network is first convolved through the convolution layer. After the convolved features pass through the two branches of the classifier and the regressor, the position of the foreground can be finally obtained.

[0043] In a specific implementation, in the process of inputting the first feature map and outputting the dynamic region in the region candidate network, the output V(F) of the region candidate network is first normalized to obtain a normalized feature map:

[0044] Map = σ(V(F));

[0045] Next, threshold processing is performed on the normalized feature map to generate the dynamic area:

[0046]

[0047] Among them, σ(·) represents the normalization function, WeightedMap x,y represents the weight of the dynamic area at the coordinate (x, y), and threshold represents the area threshold. Specifically, WeightedMap x,yThe positions where the value in is greater than zero belong to the dynamic area of ​​distillation, and the larger the value, the greater the confidence in the dynamic area. In this way, during the model distillation process, the student network can better learn the features in the teacher network. x,y The positions where the value in is equal to zero are usually noise and do not belong to the dynamic area of ​​distillation, which can effectively prevent the student network from learning the noise.

[0048] It is easy for those skilled in the art to understand that, in the example where the region candidate network is RPN, the output V(F) of the region candidate network is expressed as RPN(F).

[0049] This application normalizes the output of the region candidate network to obtain a normalized feature map, and performs threshold processing on the normalized feature map to generate a dynamic region, thereby distinguishing the foreground from the background, providing technical support for the subsequent determination of the distillation loss function. In addition, the normalized processing can further improve the distillation effect and enhance the training accuracy of the student network.

[0050] Step S102: Input the traffic scene training image into the student network to be trained to obtain a second feature map and a detection loss part, wherein the student network includes a second feature pyramid network, the second feature pyramid network outputs a second feature map, and the student network obtains the detection loss part based on the true label.

[0051] Specifically, Figure 2 As shown, the student network includes a backbone network, a second feature pyramid network and a prediction network, wherein the second feature map output by the second feature pyramid network passes through the prediction network and outputs a prediction result, and the prediction result and the true label (ground truth) can be used to determine the detection loss part of the student network.

[0052] In addition, the second feature map output by the second feature pyramid network is also used to calculate the distillation loss part.

[0053] Step S103: determining the distillation loss portion based on the first characteristic graph, the second characteristic graph and the dynamic area.

[0054] Specifically, the distillation loss part can use the square loss function (Squered L2 loss) and the maximum mean difference loss function (mmd loss) to calculate the distillation loss part.

[0055] In one embodiment of the present application, this step includes: using the dynamic area to weight the distillation loss function, using the dynamic area as a weight to constrain the first feature map to be consistent with the second feature map, thereby determining the distillation loss part.

[0056] In one implementation, taking the square loss function as an example, the distillation loss function is weighted using the dynamic region, including:

[0057]

[0058] Among them, LOSS represents the weighted square loss function, F t,x,y Indicates the feature of the first feature map at the coordinate (x, y), F s,x,y Indicates the feature of the second feature map at the coordinate (x, y), WeightedMap x,y Represents the weight of the dynamic area at the coordinate (x, y), and N is the product of the length and width of the second feature map.

[0059] This application mainly uses dynamic regions to weight the distillation loss function. The network optimization is more powerful at positions with large weights, and no optimization is performed at positions with zero weights. This improves the learning ability of the student network.

[0060] Compared with the existing technology that directly determines the loss function through the feature maps output by the teacher network and the student network, the dynamic area is used as the weight to guide the student network for distillation, which improves the student network's ability to fit the teacher network and avoids the student network's fitting of background noise, thereby enabling the training of a high-precision student network.

[0061] Step S104: Train the student network based on the distillation loss part and the detection loss part.

[0062] In one embodiment, in the process of training the student network based on the distillation loss part and the detection loss part, the sum of the distillation loss part and the detection loss part can be first used as the global loss of the student network, and then the network parameters of the student network are adjusted based on the global loss (for example, using back propagation) until the global loss meets the preset conditions, that is, the global loss converges, and the training of the student network is completed.

[0063] Based on the above steps S11-S12, the acquired image to be detected is input into the trained student network to obtain the target detection object. In the process of training the student network, the distillation loss part is determined by obtaining the dynamic area, the first feature map from the teacher network and the second feature map obtained from the student network, and then the student network is distilled using the distillation loss part and the detection loss part corresponding to the student network. In this way, the learning ability of the student network on the teacher network is increased, the distillation effect is improved, and the trained student network has higher accuracy. When it is used for target detection in the field of autonomous driving, it is beneficial to improve the detection accuracy of the target object detected, thereby achieving the effect of safe driving.

[0064] In one embodiment, before determining the distillation loss part based on the first feature map, the second feature map and the dynamic area, the method of the present application further includes: inputting the second feature map into the FitNet network so that the second feature map is aligned with the first feature map, such as Figure 2 shown.

[0065] Generally speaking, due to the different network structures of the teacher network and the student network, the height, width, and number of channels of the feature pyramid network output feature maps in the teacher network and the student network may be different. Therefore, the FitNet network is needed to align the two feature maps to further improve the distillation effect of the model.

[0066] Specifically, the FitNet network can be composed of multiple convolutional layers or deconvolutional layers connected in sequence. By setting parameters such as the number of channels and step size of the convolutional layer, the FitNet network can adjust the size of the second feature map output by the student network, thereby further improving the distillation effect of the model.

[0067] The FitNet network is used to align the second feature map with the first feature map, and the distillation loss part is calculated based on the aligned second feature map and the first feature map and the dynamic area. Finally, the distillation loss part and the detection loss part are combined to train the student network, thereby improving the distillation effect of the student model.

[0068] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0069] In addition, it should be noted that the above takes the autonomous driving scenario as an example, however, the present application is not limited to this, and the target detection method provided in the present application is also applicable to other artificial intelligence target detection scenarios.

[0070] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present invention can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0071] Furthermore, the present invention also provides an electronic device, such as Figure 4 As shown, in an electronic device embodiment according to the present invention, the electronic device includes a processor 70 and a storage device 71, the storage device can be configured to store a program for executing the knowledge distillation method or the target detection method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the knowledge distillation method or the target detection method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention.

[0072] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the knowledge distillation method or target detection method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned knowledge distillation method or target detection method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0073] Furthermore, the present invention also provides a vehicle, comprising the aforementioned electronic device. The vehicle comprising the electronic device in the present application can accurately identify the locations of targets such as lane lines, traffic lights, obstacles, and pedestrians.

[0074] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A target detection method, characterized in that: The steps include: Obtain an image to be detected containing a target detection object; Inputting the image to be detected into a student network trained by a knowledge distillation method to obtain a target detection object; wherein training the student network by using the knowledge distillation method includes: Inputting a traffic scene training image into a trained teacher network to obtain a first feature map and a dynamic region, wherein the teacher network includes a first feature pyramid network and a region candidate network, the first feature pyramid network outputs the first feature map, and the region candidate network inputs the first feature map and outputs the dynamic region, including: Normalizing the output of the region candidate network to obtain a normalized feature map; Performing threshold processing on the normalized feature map to generate the dynamic area; Inputting a traffic scene training image into a student network to be trained to obtain a second feature map and a detection loss part, wherein the student network includes a second feature pyramid network and a prediction network, the second feature pyramid network outputs the second feature map, the prediction network inputs the second feature map and outputs a prediction result, and the student network obtains the detection loss part based on the true label and the prediction result; Determining a distillation loss portion based on the first feature map, the second feature map, and the dynamic region includes: weighting a distillation loss function using the dynamic region, wherein the distillation loss function is used to constrain the first feature map and the second feature map to remain consistent; The student network is trained based on the distillation loss part and the detection loss part.

2. The target detection method according to claim 1, characterized in that: The output V(F) of the region candidate network is normalized to obtain a normalized feature map: Map = σ(V(F)); The normalized feature map is thresholded to generate the dynamic region: Among them, σ(·) represents the normalization function, WeightedMap x,y It represents the weight of the dynamic area at the coordinate (x, y), and threshold represents the area threshold.

3. The target detection method according to claim 1, characterized in that: The use of dynamic regions to weight the distillation loss function includes: Among them, LOSS represents the weighted square loss function, F t,x,y Indicates the feature of the first feature map at the coordinate (x, y), F s,x,y Indicates the feature of the second feature map at the coordinate (x, y), WeightedMap x,y represents the weight of the dynamic area at the coordinate (x, y), and N is the product of the length and width of the second feature map.

4. The target detection method according to claim 1, characterized in that: Training the student network based on the distillation loss part and the detection loss part includes: The sum of the distillation loss part and the detection loss part is used as the global loss of the student network; The network parameters of the student network are adjusted based on the global loss until the global loss meets a preset condition.

5. The target detection method according to claim 1, characterized in that: Before determining the distillation loss part based on the first characteristic map, the second characteristic map and the dynamic area, the method further includes: The second feature map is input into the FitNet network so that the second feature map is aligned with the first feature map.

6. The target detection method according to any one of claims 1 to 5, characterized in that: The region candidate network is any one of RPN, R-CNN and faster-R-CNN.

7. An electronic device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the target detection method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the target detection method according to any one of claims 1 to 6.

9. A vehicle, characterized in that: The vehicle comprises the electronic device according to claim 7.

Images