Target detection method, device and storage medium

Abstract

The embodiment of the present application provides a target detection method, device and storage medium. The method includes: performing feature extraction processing on an input image to obtain a multi-layer feature map, the feature map contains road feature information, and the downsampling rate of the multi-layer feature map is different; performing fusion processing on the multi-layer feature map to obtain multiple two-dimensional instance features corresponding to the multi-layer feature map; performing prediction processing based on the multiple two-dimensional instance features to obtain a road obstacle target in the input image. Target prediction is performed based on the feature map containing road feature information, and the accuracy of target detection can be improved by taking into account the road context information.

Description

Technical Field

[0001] The present application relates to the field of detection technology, and in particular to a target detection method, device and storage medium. Background Art

[0002] With the continuous development of autonomous driving technology, the application scenarios of autonomous driving have also expanded from simple specific scenarios such as airports and ports to the open world, such as urban roads and highways, etc. This has also brought greater challenges to driving safety.

[0003] Among the unsafe factors in the open world, non-whitelist obstacles on the road are one of the most serious threats to driving safety. If non-whitelist obstacles on the road can be detected in advance, early warning can be given, and lane changes or braking actions can be made, which can greatly improve driving safety.

[0004] However, non-whitelist obstacle detection requires extremely high preparedness. If a false alarm causes sudden braking, it is likely to cause a rear-end collision or other accidents. In actual deployment, it is usually required that the number of false detections per 100 kilometers is less than one.

[0005] Currently, two-dimensional (2D) target detection usually uses the common two-stage detector FasterRCNN or one-stage detector Yolov3, etc. The detector method uses a simple network structure and is more friendly to the detection of whitelist obstacles.

[0006] However, this detection method is difficult to solve the problem of detecting non-whitelist obstacles; and it has many false detections and it is difficult to ensure accuracy. Summary of the invention

[0007] The present application discloses a target detection method, device and storage medium, which can realize the detection of non-whitelist obstacles and improve the accuracy of target detection.

[0008] In a first aspect, an embodiment of the present application provides a target detection method. The method may include: performing feature extraction processing on an input image to obtain a multi-layer feature map. The feature map contains road feature information. The feature map also contains obstacle information. The downsampling rates of the multi-layer feature map are different. Then, each layer of the feature map in the multi-layer feature map is fused with the feature map of the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map. For the first layer of feature map, if the feature map of the previous layer does not exist, no fusion is performed. Finally, prediction processing is performed based on the multiple two-dimensional instance features to obtain the road obstacle target in the input image.

[0009] The different sampling rates may be different scales of feature maps. The larger the downsampling rate, the smaller the feature map size of the layer. For example, the downsampling rates are 8 times, 16 times, 32 times, and 64 times, respectively. Among them, the size of the feature map of 8 times downsampling is 1 / 8 of the input image.

[0010] In the embodiment of the present application, a feature extraction process is performed on the input image to obtain multiple layers of feature maps containing road feature information; the multiple layers of feature maps are then fused to obtain multiple two-dimensional instance features; and road obstacle targets are obtained based on the multiple two-dimensional instance features. By adopting this method, target prediction is performed based on the feature map containing road feature information, and the accuracy of target detection can be improved by taking into account the road context information.

[0011] In a possible implementation, the method further includes: performing processing according to the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask. Then, performing prediction processing according to the multiple two-dimensional instance features and the road segmentation mask to obtain a road obstacle target in the input image.

[0012] This example performs prediction processing based on multiple two-dimensional instance features and the road segmentation mask, which can further reduce the false detection rate.

[0013] In a possible implementation, a plurality of road surface features respectively corresponding to the multi-layer feature map are obtained based on the multi-layer feature map and the plurality of two-dimensional instance features. For the i-th layer feature map, a fusion process is performed based on the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain a fused feature. Then, the fused feature is fused with the two-dimensional instance features corresponding to the i-th layer feature map to obtain the road surface features corresponding to the i-th layer feature map. The plurality of road surface features respectively corresponding to the multi-layer feature map include the road surface features corresponding to the i-th layer feature map, and i is an integer not less than 1. When i is 1, the road surface features corresponding to the 0th layer feature map do not exist. Then, the road surface segmentation mask is obtained based on the plurality of road surface features respectively corresponding to the multi-layer feature map.

[0014] Among them, the road segmentation mask is a binary image of the road surface prediction. Based on the above fusion processing, multiple road surface features can be obtained, and then the road surface segmentation mask can be obtained.

[0015] In a possible implementation, prediction processing is performed based on the multiple two-dimensional instance features to obtain an initial prediction box. It is determined whether the center point of the initial prediction box is on the road surface based on the road surface segmentation mask. If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

[0016] This example filters obstacles that are not on the road based on the road segmentation mask, which can further reduce the false detection rate.

[0017] In a possible implementation, the method further includes: performing segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features respectively corresponding to the multi-layer feature map. Then, performing prediction processing on the input image according to the plurality of two-dimensional instance features and the plurality of obstacle features to obtain a road obstacle target in the input image.

[0018] By adopting this scheme, processing is performed based on the feature map containing road feature information and combining the obtained obstacle features, the category-independent features of the target area can be enhanced to enhance the versatility of obstacles and further improve the accuracy of target detection.

[0019] In a possible implementation, prediction processing is performed based on the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level. It is determined whether the center point of the initial prediction box falls within the obstacle based on the multiple obstacle features. If the center point of the initial prediction box falls within the obstacle, the first confidence level is updated to a second confidence level, and the second confidence level is greater than the first confidence level. If the second confidence level is greater than a preset value, the road obstacle target in the input image is determined based on the initial prediction box.

[0020] This example performs matching based on a preset confidence value of the obstacle feature and a confidence obtained based on the two-dimensional instance feature, which can further reduce the false detection rate.

[0021] In one possible implementation, the initial prediction box is used as the road obstacle target in the input image, or, it is determined whether the center point of the initial prediction box is on the road surface based on a road segmentation mask, where the road segmentation mask is obtained by processing the multi-layer feature map and the multiple two-dimensional instance features; if the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

[0022] Through double confirmation of road segmentation mask and obstacle features, obstacles that are not on the road are filtered out, and matching is performed based on the preset confidence value of the obstacle feature and the confidence obtained based on the two-dimensional instance feature, which can further reduce the false detection rate.

[0023] In a possible implementation, the prediction model includes a backbone network and a main detection branch network, the main detection branch network includes a neck network and a head network; feature extraction processing of the input image to obtain the multi-layer feature map is implemented by the backbone network, and each layer of the feature map in the multi-layer feature map is fused with the previous layer of the feature map to obtain multiple two-dimensional instance features corresponding to the multi-layer feature map respectively is implemented by the neck network; prediction processing based on the multiple two-dimensional instance features to obtain the road obstacle target in the input image is implemented by the head network.

[0024] In this example, the above target detection is implemented in the prediction model.

[0025] In a possible implementation, the backbone network and the neck network are trained in the following manner: for the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing in to obtain a multi-layer sample feature map of the sample image, where k is an integer not less than 1; input the multi-layer sample feature map into the neck network N k The fusion process is performed to obtain a plurality of two-dimensional instance features corresponding to the multi-layer sample feature map; a predicted value of the road segmentation mask is obtained according to the plurality of two-dimensional instance features and the multi-layer sample feature map; a loss value is calculated according to the labeled value and the predicted value, and a gradient is calculated according to the loss value; the backbone network Z is adjusted according to the gradient k and the neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

[0026] Based on this training, a backbone network and a neck network that can extract road feature information can be obtained.

[0027] In a second aspect, the present application provides a target detection device. The device includes: a processing module, which is used to perform feature extraction processing on an input image to obtain a multi-layer feature map, the feature map contains road feature information, and the downsampling rate of the multi-layer feature map is different. The processing module is also used to fuse each layer of the feature map in the multi-layer feature map with the previous layer of the feature map to obtain multiple two-dimensional instance features corresponding to the multi-layer feature map. A prediction module is used to perform prediction processing based on the multiple two-dimensional instance features to obtain a road obstacle target in the input image.

[0028] In a possible implementation, the processing module is further used to: perform processing according to the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask. The prediction module is further used to: perform prediction processing according to the multiple two-dimensional instance features and the road segmentation mask to obtain a road obstacle target in the input image.

[0029] In a possible implementation, the processing module is further used to: perform processing according to the multi-layer feature map and the multiple two-dimensional instance features to obtain multiple road surface features respectively corresponding to the multi-layer feature map; wherein, for the i-th layer feature map, perform fusion processing according to the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain fused features; perform fusion processing on the fused features and the two-dimensional instance features corresponding to the i-th layer feature map to obtain the road surface features corresponding to the i-th layer feature map, wherein the multiple road surface features respectively corresponding to the multi-layer feature map include the road surface features corresponding to the i-th layer feature map, i is an integer not less than 1, and when i is 1, the road surface features corresponding to the 0th layer feature map do not exist; and obtain the road surface segmentation mask according to the multiple road surface features respectively corresponding to the multi-layer feature map.

[0030] In a possible implementation, the prediction module is further used to: perform prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction frame. Determine whether the center point of the initial prediction frame is on the road surface according to the road surface segmentation mask. If the center point of the initial prediction frame is on the road surface, use the initial prediction frame as a road obstacle target in the input image.

[0031] In a possible implementation, the processing module is further used to: perform segmentation mask prediction processing on the multi-layer feature map to obtain multiple obstacle features corresponding to the multi-layer feature map. The prediction module is further used to: perform prediction processing based on the multiple two-dimensional instance features and the multiple obstacle features to obtain the road obstacle target in the input image.

[0032] In a possible implementation, the prediction module is further used to: perform prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level. Determine whether the center point of the initial prediction box falls within the obstacle according to the obstacle feature. If the center point of the initial prediction box falls within the obstacle, update the first confidence level to a second confidence level, and the second confidence level is greater than the first confidence level. If the second confidence level is greater than a preset value, determine the road obstacle target in the input image according to the initial prediction box.

[0033] In a possible implementation, the prediction module is further used to: use the initial prediction box as a road obstacle target in the input image. Alternatively, determine whether the center point of the initial prediction box is on the road surface according to a road segmentation mask, wherein the road segmentation mask is obtained by processing the multi-layer feature map and the multiple two-dimensional instance features. If the center point of the initial prediction box is on the road surface, use the initial prediction box as a road obstacle target in the input image.

[0034] In a possible implementation, the processing module includes a backbone network and a main detection branch network, the main detection branch network includes a neck network and a head network; feature extraction processing of the input image to obtain the multi-layer feature map is implemented by the backbone network, and each layer of the feature map in the multi-layer feature map is fused with the previous layer of the feature map to obtain multiple two-dimensional instance features corresponding to the multi-layer feature map is implemented by the neck network; prediction processing based on the multiple two-dimensional instance features to obtain the road obstacle target in the input image is implemented by the head network.

[0035] In a possible implementation, the backbone network and the neck network are trained in the following manner: for the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing in to obtain a multi-layer sample feature map of the sample image, where k is an integer not less than 1; input the multi-layer sample feature map into the neck network N k The fusion process is performed to obtain a plurality of two-dimensional instance features corresponding to the multi-layer sample feature map; a predicted value of the road segmentation mask is obtained according to the plurality of two-dimensional instance features and the multi-layer sample feature map; a loss value is calculated according to the labeled value and the predicted value, and a gradient is calculated according to the loss value; the backbone network Z is adjusted according to the gradient k and the neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

[0036] In a third aspect, the present application provides a target detection device, which includes a processor and a communication interface, the communication interface is used to receive and / or send data, and / or, the communication interface is used to provide output and / or output to the processor, and the processor is used to call computer instructions to implement a method as described in any one of the first aspects.

[0037] In a possible implementation manner, the target detection device further includes the one or more memories.

[0038] In a possible implementation, the target detection device is a chip or a chip system.

[0039] In a fourth aspect, the present application provides a computer storage medium comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute a method as provided in any possible implementation of the first aspect.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute a method provided in any possible implementation manner of the first aspect.

[0041] It can be understood that the device described in the second aspect, the device described in the third aspect, the computer storage medium described in the fourth aspect, or the computer program product described in the fifth aspect provided above are all used to execute any method provided in the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The following is an introduction to the drawings used in the embodiments of the present application.

[0043] Figure 1 is a schematic diagram of a target detection system applicable to an embodiment of the present application;

[0044] Figure 2 It is a flowchart of a target detection method provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a feature extraction network provided in an embodiment of the present application;

[0046] Figure 4 is a flow chart of another target detection method provided in an embodiment of the present application;

[0047] Figure 5a It is a schematic diagram of a model processing provided in an embodiment of the present application;

[0048] Figure 5b It is a schematic diagram of a model provided in an embodiment of the present application;

[0049] Figure 6 It is a flowchart of another target detection method provided in an embodiment of the present application;

[0050] Figure 7 It is another model processing schematic diagram provided in an embodiment of the present application;

[0051] Figure 8 It is another model processing schematic diagram provided in the embodiment of the present application;

[0052] Fig. 9 This is another model schematic diagram provided in the embodiment of the present application;

[0053] Fig.10 is a structural schematic diagram of a target detection device provided in an embodiment of the present application;

[0054] Fig.11 It is a structural schematic diagram of another target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0056] For ease of understanding, the following examples provide some explanations of concepts related to the embodiments of the present application for reference. As described below:

[0057] 1. Whitelist obstacles: obstacles pre-defined in the whitelist, such as people, motor vehicles, non-motor vehicles, traffic cones, traffic poles, water barriers, etc. on the road.

[0058] 2. Non-whitelist obstacles: refers to all possible obstacles on the road that are not in the whitelist, such as broken tires, stones, cardboard boxes, garbage bags, etc.

[0059] The above exemplary description of the concepts can be applied in the following embodiments.

[0060] Currently, common two-stage detectors such as FasterRCNN or one-stage detectors such as Yolov3 are used to detect whitelist obstacles. However, it is difficult to solve the detection of non-whitelist obstacles; and there are many false detections, making it difficult to ensure accuracy. In view of this, the present application provides a target detection method, device, and storage medium, which can realize the detection of non-whitelist obstacles and improve the accuracy of target detection.

[0061] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Figure 1 , Figure 1 1 is a schematic diagram of a target detection system applicable to an embodiment of the present application, the system includes a vehicle 101 and a service end 102. Among them:

[0062] The vehicle 101 is a device with communication and computing capabilities, and can provide mobile travel services for users. The vehicle 101 can provide an environment for deploying software, hardware, or a combination of software and hardware modules. For example, software can be installed on the vehicle 101. For another example, the vehicle 101 has an interface for connecting hardware, and the hardware can be connected to the vehicle 101 through the interface. For another example, the vehicle 101 has an environment for installing hardware drivers.

[0063] The server 102 is a device with centralized computing capabilities. Exemplarily, the server 102 can be implemented by a server, a virtual machine, a cloud, a roadside device, or a robot.

[0064] When the server 102 includes a server, the type of the server includes but is not limited to a general-purpose computer, a dedicated server computer, a blade server, etc. This application does not impose a strict limit on the number of servers included in the server 102, and the number can be one or more (such as a server cluster, etc.).

[0065] A virtual machine refers to a computing module that has complete hardware system functions and runs in a completely isolated environment through software simulation. Of course, in addition to virtual machines, the server 102 can also be implemented through other computing instances, such as containers.

[0066] The cloud is a software platform that uses application virtualization technology, which enables one or more software and applications to be developed and run in an independent virtualized environment. Optionally, when the server 102 is implemented through the cloud, the cloud can be deployed on a public cloud, a private cloud, or a hybrid cloud.

[0067] A roadside device is a device installed on the side of a road (or at an intersection, or on the side of a road, etc.). The road can be an outdoor road (such as a main road, a secondary road, an elevated road, or a temporary road, etc.), or an indoor road (such as a road in an indoor parking lot, etc.). A roadside device can provide services for vehicles. It should be noted that a roadside device can be an independent device or integrated into other devices. For example, a roadside device can be integrated into devices such as smart gas stations, charging piles, smart traffic lights, street lights, telephone poles, or traffic signs.

[0068] Since there may be some obstacles on the road where the vehicle 101 is traveling, such as people, motor vehicles, non-motor vehicles, traffic cones, traffic poles, water barriers, broken tires, stones, cartons, garbage bags, etc., there may be some safety hazards when the vehicle 101 is traveling. For example, in scenes such as automatic driving and assisted driving, these obstacles need to be detected to better guide the vehicle 101 to travel.

[0069] In the embodiment of the present application, the server 102 can perform feature extraction processing based on the road image to obtain a multi-layer feature map, which contains road feature information. The server 102 obtains multiple two-dimensional instance features corresponding to the multi-layer feature map by fusing the multi-layer feature map; then, the server 102 performs prediction processing based on the multiple two-dimensional instance features to obtain the road obstacle target in the road image.

[0070] The target detection method provided in the embodiment of the present application can improve the detection rate and accuracy of the target.

[0071] The embodiments of the present application can be applied to the visual perception system of an Advanced Driving Assistant System (ADAS) or an Autonomous Driving System (ADS). It can also be applied to vehicle-mounted visual perception equipment and security and other perception equipment, which is not strictly limited by the present solution.

[0072] The architecture of the embodiment of the present application is described above, and the method of the embodiment of the present application is described in detail below.

[0073] Reference Figure 2 FIG. 1 is a flow chart of a target detection method provided in an embodiment of the present application. Optionally, the method can be applied to the aforementioned target detection system, for example Figure 1 The target detection system shown in Figure 1 is Figure 2 The target detection method shown may include steps 201-203. It should be understood that for the convenience of description, this application is described in the order of 201-203, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 201-203 of the target detection method as a server as an example, and this application is also applicable to other execution subjects. Steps 201-203 are as follows:

[0074] 201. Perform feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map includes road feature information, and the multi-layer feature map has different downsampling rates;

[0075] The input image is an image containing a road. The input image may be sent by a vehicle or may be obtained by the server itself, such as from a roadside device, etc. This solution does not limit this.

[0076] The road surface feature information can be understood as the context information of the road surface, for example, it may include the texture and color of the road surface.

[0077] Optionally, the feature map also includes obstacle information.

[0078] The feature extraction process may be, for example, a convolution process, or a Transformer process, or a combination of convolution and Transformer.

[0079] The downsampling rates of the above-mentioned multi-layer feature maps are different, that is, the scales of the multi-layer feature maps are different.

[0080] By processing the input image, a multi-layer feature map containing road feature information can be obtained. Since the feature map contains road feature information, it helps to improve the detection accuracy of obstacle targets in subsequent target detection processing.

[0081] In a possible implementation, the multi-layer feature map is obtained by inputting the input image into a feature extraction network for processing.

[0082] Optionally, the feature extraction network may include a backbone network. The backbone network may be in various forms, such as a Visual Geometry Group (VGG) network, a residual network Resnet, an Inception-net, etc.

[0083] Alternatively, the feature extraction network includes a backbone network and a feature pyramid network (FPN). For example, the architecture of the backbone network and the feature pyramid network in the feature extraction network is as follows: Figure 3 As shown in the figure, backbone performs a series of convolution or Transformer processing on the input image to obtain features at different scales (such as C3, C4, C5, etc.). These features can be used as the basic features for subsequent target detection processing.

[0084] FPN can generate more expressive feature maps (such as P3, P4, P5, P6, etc.) through its internal vertical feature fusion and fusion with the horizontal features of the same layer of the backbone (such as C3, C4, C5, etc.) for subsequent target detection processing. For example, the downsampling rates of the above feature maps P3, P4, P5, and P6 are 8 times, 16 times, 32 times, and 64 times, respectively. Among them, the above multi-layer feature maps can be understood as feature maps of different downsampling layers. The larger the downsampling rate, the smaller the size of the feature map of the layer. For example, the size of the feature map downsampled 8 times is 1 / 8 of the input image.

[0085] 202. Fusing each layer of the feature maps in the multi-layer feature maps with the feature maps in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature maps;

[0086] Two-dimensional 2D instance features, that is, 2D features of various objects in the feature map.

[0087] Since the above-mentioned feature map includes road surface feature information, the two-dimensional instance feature also includes road surface feature information.

[0088] By fusing each layer of the feature maps in the above multi-layer feature maps, more robust features can be obtained. The fusion process can be to fuse the feature map of the previous layer with the feature map of the current layer to obtain the two-dimensional instance features corresponding to the current layer. By analogy, multiple two-dimensional instance features corresponding to the multi-layer feature maps can be obtained. For the first layer of feature maps, if the feature map of the previous layer does not exist, no fusion is performed.

[0089] In a possible implementation, by inputting the above multi-layer feature map into a 2D neck network for processing, a plurality of two-dimensional instance features corresponding to the multi-layer feature map can be obtained. The 2D neck network is a convolutional network composed of a plurality of residual networks cascaded, which is used to enhance the learning ability of the network and make the features more robust.

[0090] 203. Perform prediction processing according to the multiple two-dimensional instance features to obtain a road obstacle target in the input image.

[0091] The prediction process can be understood as the process of converting multiple two-dimensional instance features into detection boxes.

[0092] In a possible implementation, by inputting the above-mentioned multiple two-dimensional instance features into the 2D head network for prediction, the detection box on each layer of the feature map can be output. The 2D head network is composed of a cascade of several convolutional layers, which is used to predict the bounding box and confidence of the obstacle based on each point on the feature map.

[0093] Then, the final detection frame can be obtained based on the detection frames on the multi-layer feature map, and the position corresponding to the final detection frame is the position of the road obstacle target in the input image. For example, the final detection frame can be obtained by combining the detection frames on each layer of the feature map, removing overlapping detection frames through non-maximum suppression, and filtering out detection frames less than the confidence threshold.

[0094] The road obstacle target may be a non-whitelist obstacle or a whitelist obstacle, and this solution does not impose any restrictions on this.

[0095] By adopting this solution, the road obstacle target obtained by processing based on the feature map containing road feature information has a high accuracy.

[0096] In a possible implementation, before step 203, the method further includes:

[0097] Processing is performed according to the multi-layer feature map and the multiple two-dimensional instance features to obtain a road surface segmentation mask.

[0098] The road segmentation mask is a binary image of the road surface prediction.

[0099] For example, the multi-layer feature map is subjected to road surface context information fusion processing according to the multi-layer feature map and the multiple two-dimensional instance features to obtain multiple road surface features respectively corresponding to the multi-layer feature map. Then, the road surface segmentation mask is obtained according to the multiple road surface features respectively corresponding to the multi-layer feature map.

[0100] For the i-th layer feature map, a fusion process is performed according to the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain fused features.

[0101] The fused features and the two-dimensional instance features corresponding to the i-th layer feature map are fused again to obtain the road surface features corresponding to the i-th layer feature map. Repeating the above steps, multiple road surface features corresponding to multiple layers of feature maps can be obtained.

[0102] Wherein, i is an integer not less than 1. When i is 1, the road surface feature corresponding to the feature map of layer 0 does not exist. In other words, when the feature map of layer 1 is fused, only the feature map of layer 1 and the two-dimensional instance features corresponding to the feature map of layer 1 are fused to obtain the road surface feature corresponding to the feature map of layer 1.

[0103] Based on the multiple road features obtained above, the road segmentation mask can be obtained by taking the last road feature for classification prediction.

[0104] Accordingly, step 203 may include:

[0105] Prediction processing is performed based on the multiple two-dimensional instance features and the road segmentation mask to obtain the road obstacle target in the input image.

[0106] For example, prediction processing is performed respectively according to the multiple two-dimensional instance features to obtain multiple detection frames. Then, overlapping detection frames are removed through non-maximum suppression, and detection frames with a value less than a confidence threshold are filtered out to obtain an initial prediction frame.

[0107] Then, it is determined whether the center point of the initial prediction box is on the road surface according to the initial prediction box and the road surface segmentation mask. If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road surface obstacle target in the input image. That is, the object in the initial prediction box is the road surface obstacle target.

[0108] If the center point of the initial prediction frame is not on the road surface, it means that the initial prediction frame is not a road obstacle target, and the initial prediction frame is discarded and regarded as an invalid detection prediction.

[0109] This example filters obstacles that are not on the road based on the road segmentation mask, which can further reduce the false detection rate.

[0110] In the embodiment of the present application, multiple feature maps containing road feature information are obtained by performing feature extraction processing on the input image; multiple two-dimensional instance features are then obtained by performing fusion processing on the multi-layer feature maps; and road obstacle targets are obtained based on the multiple two-dimensional instance features. By adopting this method, target prediction is performed based on the feature map containing road feature information, and the accuracy of target detection can be improved by taking into account the road context information.

[0111] Among them, in addition to the accuracy problem, the detection of non-whitelist obstacles also faces the following challenges: Since the categories of non-whitelist obstacles are not clearly defined, but any objects that may appear on the road that are not in the whitelist obstacles. It is unrealistic to simply enumerate all possible objects. There are always some obstacles that appear unexpectedly on the road, such as TVs, refrigerators, etc. Therefore, when performing target detection, it is necessary for the server and other devices to be able to understand "abnormal objects on the road" to solve the universality of non-whitelist obstacles.

[0112] Based on this, the embodiment of the present application also provides another target detection method. Figure 4 As shown, this method can be applied to the aforementioned target detection system, such as Figure 1 The target detection system shown in Figure 1 is Figure 4 The target detection method shown may include steps 401-404. It should be understood that for the convenience of description, this application is described in the order of 401-404, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, number of executions, etc. of the above one or more steps. The following description takes the execution subject of steps 401-404 of the target detection method as a server as an example, and this application is also applicable to other execution subjects. Steps 401-404 are as follows:

[0113] 401. Perform feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map includes road feature information, and the multi-layer feature map has different downsampling rates;

[0114] For an introduction to this part, please refer to the above Figure 2 The introduction of step 201 in the illustrated embodiment will not be repeated here.

[0115] 402. Fusing each layer of the feature maps in the multi-layer feature maps with the feature maps in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature maps;

[0116] For an introduction to this part, please refer to the above Figure 2 The introduction of step 202 in the illustrated embodiment will not be repeated here.

[0117] 403. Perform segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features corresponding to the multi-layer feature map respectively;

[0118] For example, prediction can be performed by using convolutional layers and activation layers to obtain multiple obstacle features (i.e., obstacle segmentation masks).

[0119] By obtaining obstacle features, it is possible to enhance the category-independent general features of the target area (i.e., the area corresponding to the obstacle target in the input image), and correspondingly reduce the response of the non-target area, thereby reducing false detection and improving the accuracy of target detection.

[0120] In a possible implementation, the multi-layer feature map is input into an obstacle feature guidance module for processing, so as to obtain a plurality of obstacle features respectively corresponding to the multi-layer feature map.

[0121] For example, Figure 5a As shown, the obstacle feature guidance module uses a mask attention (MAM) module to perform prediction processing on the feature map based on each feature layer to obtain a foreground segmentation mask.

[0122] Among them, the structure of the MAM module is as follows Figure 5b As shown in the figure, after the feature map output by the aforementioned FPN passes through the convolution layer and the activation layer sigmoid, the output dimension is a mask of 1xHxW, where H and W are the height and width of the feature map respectively.

[0123] 404. Perform prediction processing according to the multiple two-dimensional instance features and the multiple obstacle features to obtain a road obstacle target in the input image.

[0124] In a possible implementation manner, a fusion process is performed based on the multiple two-dimensional instance features and the multiple obstacle features to obtain a fusion feature of a common feature of a non-whitelist obstacle with road surface context information.

[0125] The fused features are then input into the 2D head network for processing to obtain the initial prediction frame, that is, the road obstacle target.

[0126] By adopting this scheme, processing is performed based on the feature map containing road feature information and combining the obtained obstacle features, the category-independent features of the target area can be enhanced to enhance the versatility of obstacles and further improve the accuracy of target detection.

[0127] For example, Figure 5a As shown in FIG. 1 , each obstacle feature, namely the obstacle segmentation mask, is fused with the corresponding two-dimensional instance feature to obtain a fused feature. The fused feature is then input into the 2D head network for processing to obtain a road obstacle target.

[0128] Optionally, prediction processing is performed based on the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level.

[0129] It is determined whether the center point of the initial prediction box is in the obstacle segmentation mask according to the obstacle feature (obstacle segmentation mask).

[0130] If the center point of the initial prediction box is in the obstacle segmentation mask, the first confidence is updated to a second confidence, and the second confidence is greater than the first confidence. For example, the first confidence is multiplied by a coefficient greater than 1 to obtain the second confidence. If the second confidence is greater than a preset value, the road obstacle target in the input image is determined according to the initial prediction box.

[0131] If the center point of the initial prediction box is not in the obstacle segmentation mask, the first confidence is updated to a third confidence, and the third confidence is less than the first confidence. For example, the first confidence is multiplied by a coefficient less than 1 to obtain the third confidence. If the third confidence is greater than the filter preset value, the road obstacle target in the input image is determined based on the initial prediction box. If the third confidence is not greater than the filter preset value, the initial prediction box is discarded and regarded as an invalid detection prediction.

[0132] This example performs matching based on a preset confidence value of the obstacle segmentation mask and a confidence obtained based on two-dimensional instance features, which can further reduce the false detection rate.

[0133] In the embodiment of the present application, multiple feature maps containing road feature information are obtained by performing feature extraction processing on the input image, and multiple two-dimensional instance features and multiple obstacle features are obtained based on the multi-layer feature map; and then the road obstacle target is obtained based on the multiple two-dimensional instance features and multiple obstacle features. With this method, processing is performed based on the feature map containing road feature information, and the obtained obstacle features are combined, and the road context information is taken into account, and the category-independent features of the target area are enhanced, so as to achieve the goal of enhancing the versatility of obstacles, which can further improve the accuracy of target detection.

[0134] Based on the above embodiments, Figure 6 FIG. 1 is a flow chart of another target detection method provided in an embodiment of the present application. Optionally, the method can be applied to the aforementioned target detection system, for example Figure 1 The target detection system shown in Figure 1 is Figure 6 The target detection method shown may include steps 601-606. It should be understood that for the convenience of description, this application is described in the order of 601-606, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 601-606 of the target detection method as a server as an example, and this application is also applicable to other execution subjects. Steps 601-606 are as follows:

[0135] 601. Perform feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map includes road feature information, and the multi-layer feature map has different downsampling rates;

[0136] For the introduction of step 601, please refer to the aforementioned Figure 2 The description of step 201 in the illustrated embodiment will not be repeated here.

[0137] 602. Fusing each layer of feature maps in the multi-layer feature maps with the feature maps of the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature maps;

[0138] For the introduction of step 602, please refer to the aforementioned Figure 2 The description of step 202 in the illustrated embodiment will not be repeated here.

[0139] 603. Processing is performed according to the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask;

[0140] For the introduction of step 603, please refer to the aforementioned Figure 2 The description of step 203 in the illustrated embodiment will not be repeated here.

[0141] 604. Perform segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features corresponding to the multi-layer feature map respectively;

[0142] For the introduction of step 604, please refer to the aforementioned Figure 4 The description of step 403 in the illustrated embodiment will not be repeated here.

[0143] 605. Perform prediction processing according to the multiple two-dimensional instance features and the multiple obstacle features to obtain an initial prediction frame;

[0144] In a possible implementation manner, a fusion process is performed based on the multiple two-dimensional instance features and the multiple obstacle features to obtain a fusion feature of a common feature of a non-whitelist obstacle with road surface context information.

[0145] The fused features are then input into the 2D head network for processing to obtain the initial prediction box and the first confidence.

[0146] For an introduction to this part, please refer to the above Figure 4 The description of step 404 in the embodiment will not be repeated here.

[0147] 606. Obtain a road obstacle target in the input image according to the road segmentation mask, the multiple obstacle features, and the initial prediction box.

[0148] Optionally, the initial prediction frame is filtered based on the road segmentation mask and multiple obstacle features (obstacle segmentation mask) to filter out obstacles that are not on the road surface, which can further reduce the false detection rate.

[0149] Specifically, the above-mentioned initial prediction frame is first fused according to multiple obstacle segmentation masks to determine whether the center point of the initial prediction frame falls within the range of multiple obstacle segmentation masks. If the center point of the initial prediction frame is outside the range of the obstacle segmentation mask, the first confidence is updated to a third confidence, and the third confidence is less than the first confidence. For example, the first confidence is multiplied by a coefficient less than 1 to obtain the third confidence. If the third confidence is not greater than the filter preset value, the initial prediction frame is discarded and regarded as an invalid detection prediction.

[0150] If the center point of the initial prediction box falls within the range of the obstacle segmentation mask, the first confidence is updated to a second confidence, and the second confidence is greater than the first confidence. For example, the first confidence is multiplied by a coefficient greater than 1 to obtain the second confidence.

[0151] If the second confidence is greater than a preset value, or if the third confidence is greater than a filter preset value, it is determined whether the center point of the initial prediction box is within the road segmentation mask area. If it is within the road segmentation mask area, the output of the detection prediction is retained; otherwise, it is discarded and regarded as an invalid detection prediction.

[0152] This example readjusts the confidence of the obstacle based on the positional relationship between the obstacle segmentation mask and the initial prediction box to reduce false detections outside the foreground area. In addition, the positional relationship between the road segmentation mask and the initial prediction box is combined to ensure that the obstacle is on the road surface, thereby further reducing the false detection rate.

[0153] like Figure 7 As shown, the input image is input into the prediction model for processing to obtain the road obstacle target. The prediction model includes a backbone network and a main detection branch network, and the main detection branch network includes a neck network and a head network. Among them, the backbone network extracts features from the input image and obtains a multi-layer feature map containing road feature information. And, the obstacle guidance module processes the multi-layer feature map to obtain multiple obstacle features. Then, the neck network (such as a 2d neck network) processes the input multi-layer feature map to obtain multiple two-dimensional instance features. The head network (such as a 2dhead network) processes the multiple two-dimensional instance features and multiple obstacle features to obtain an initial prediction frame. Finally, based on the road segmentation mask obtained from the multiple two-dimensional instance features and the multi-layer feature map, as well as the multiple obstacle features and the initial prediction frame, the final prediction frame is determined as the output result.

[0154] In an embodiment of the present application, a plurality of feature maps containing road feature information are obtained by performing feature extraction processing on an input image; a plurality of two-dimensional instance features and a plurality of obstacle features are obtained based on the multi-layer feature map, and a fusion feature of a common feature of a non-whitelist obstacle with road context information can be obtained, and then an initial prediction frame is predicted based on the fusion feature, thereby improving the accuracy of target detection; and the initial prediction frame is further filtered in combination with a road segmentation mask and obstacle features to filter out obstacles that are not on the road surface, thereby further reducing the false detection rate.

[0155] The above embodiment introduces the target detection method. The following is an introduction to the prediction model of the embodiment of the present application.

[0156] By inputting the image into the prediction model for processing, the road obstacle target in the input image can be obtained. The prediction model includes a backbone network and a main detection branch network, and the main detection branch network includes a neck network and a head network.

[0157] In a possible implementation, the feature extraction processing of the input image to obtain the multi-layer feature map described in the aforementioned steps 201, 401 and 601 is implemented through the backbone network.

[0158] This example introduces the feature extraction using a backbone network. Of course, the feature extraction network may also include the aforementioned FPN, which is not limited in this solution.

[0159] For an introduction to the backbone network, please refer to the above Figure 3 The examples shown will not be described in detail here.

[0160] In a possible implementation, the steps 202, 402 and 602 described above of fusing each layer of the multi-layer feature map with the previous layer of feature map to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature maps are implemented through a neck network.

[0161] For an introduction to the neck network, see the previous Figure 2 The examples shown will not be described in detail here.

[0162] In a possible implementation, the step 203 of performing prediction processing according to the multiple two-dimensional instance features to obtain the road obstacle target in the input image is implemented through a head network.

[0163] For an introduction to the head network, see the previous Figure 2 The examples shown will not be described in detail here.

[0164] In a possible implementation, the backbone network and the neck network are trained in the following manner:

[0165] For the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing to obtain multiple sample feature maps of the sample image, where k is an integer not less than 1;

[0166] The multiple sample feature maps are input into the neck network N k Performing fusion processing in order to obtain a plurality of two-dimensional instance features respectively corresponding to the plurality of sample feature maps;

[0167] Obtaining a predicted value of a road segmentation mask according to the plurality of two-dimensional instance features and the plurality of sample feature maps;

[0168] A loss value is calculated according to the marked value and the predicted value, and a gradient is calculated according to the loss value;

[0169] Adjust the backbone network Z according to the gradient k and the neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

[0170] Reference Figure 8 As shown, it is a schematic diagram of a model processing provided in an embodiment of the present application. Figure 8 The modules shown in the dashed box can be Figure 7 The road surface feature guidance module in the main detection branch is used to generate road surface features, so that the backbone network (feature extraction module) and the neck network (2D neck network) can learn road surface feature information. This module generates road surface features by fusing the 2D instance features on different feature layers of the 2D neck in the main detection branch with the dense features in the FPN through the road surface context information fusion layer (Context Merging Layer, CML), and uses the road surface segmentation mask data for supervision during training, so that the backbone and FPN can extract road surface features and implicitly encode the context information of the road surface into the neck of the main detection branch, so that the features of the main detection branch have the context information of the road surface (i.e., road surface feature information).

[0171] The structure of the road context information fusion layer CML is as follows: Fig. 9 For each layer, the FPN dense features (i.e., the output feature map of FPN) are processed by a convolution layer and then fused with the road surface features of the previous layer through a dot product operation. The fused features are processed by a convolution layer and then fused with the 2D instance features processed by the convolution layer through a dot product operation. Finally, the road surface features of this layer are output after being processed by a batch normalization (BN) layer and a rectified linear unit (ReLu) activation layer.

[0172] Among them, dense feature A is the encoding of FPN feature map, and 2d instance feature B is the encoding of the feature output by 2dneck detection. The size scale of FPN dense feature A and 2d instance feature B is the same as that of road feature X. After the above fusion operation, the final context information fusion layer encodes the road context information into the 2d instance feature through implicit learning, which can provide more robust and rich features for subsequent target detection tasks.

[0173] The prediction model provided in the embodiment of the present application generates road features based on the road context information fusion layer CML by learning to fuse the 2d instance features on different feature layers of the 2d neck in the main detection branch with the dense features in FPN, and uses road segmentation mask data for supervision during training, so that the backbone and FPN can extract road features and implicitly encode the contextual information of the road into the neck of the main detection branch, so that the features of the main detection branch have road feature information. This can provide more robust and rich features for subsequent target detection tasks and improve the accuracy of target detection.

[0174] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0175] The above detailed description of the method of the embodiment of the present application, the following provides a device of the embodiment of the present application. It can be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division according to function, and is not used as a limitation on the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may also be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices contain a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Usually, each unit corresponds to its own program code (or program instruction), and when the program code corresponding to each of these units is run on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.

[0176] The embodiments of the present application also provide a device for implementing any of the above methods. For example, a target detection device is provided, which includes a module (or means) for implementing each step performed by the server in any of the above methods.

[0177] For example, refer to Fig.10 FIG. 1 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application. The target detection device is used to implement the aforementioned target detection method, for example Figure 2 , Figure 4 or Figure 6 The target detection method shown.

[0178] like Fig.10 As shown, the device may include a processing module 1001 and a prediction module 1002, which are specifically as follows:

[0179] The processing module 1001 is used to perform feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map contains road feature information and the downsampling rates of the multi-layer feature map are different;

[0180] The processing module 1001 is further used to fuse each layer of the feature map in the multi-layer feature map with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map;

[0181] The prediction module 1002 is used to perform prediction processing according to the multiple two-dimensional instance features to obtain the road obstacle target in the input image.

[0182] In a possible implementation, the processing module 1001 is further configured to:

[0183] Processing the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask;

[0184] The prediction module 1002 is further used for:

[0185] Prediction processing is performed based on the multiple two-dimensional instance features and the road segmentation mask to obtain the road obstacle target in the input image.

[0186] In a possible implementation, the processing module 1001 is further configured to:

[0187] According to the multi-layer feature map and the multiple two-dimensional instance features, a plurality of road surface features respectively corresponding to the multi-layer feature map are obtained;

[0188] Wherein, for the i-th layer feature map, a fusion process is performed according to the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain a fused feature;

[0189] The fused features are fused with the two-dimensional instance features corresponding to the i-th layer feature map to obtain the road surface features corresponding to the i-th layer feature map, wherein the plurality of road surface features corresponding to the multiple layers of feature maps respectively include the road surface features corresponding to the i-th layer feature map, i is an integer not less than 1, and when i is 1, the road surface features corresponding to the 0-th layer feature map do not exist;

[0190] The road surface segmentation mask is obtained according to a plurality of road surface features respectively corresponding to the multi-layer feature map.

[0191] In a possible implementation, the prediction module 1002 is further configured to:

[0192] Perform prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box;

[0193] Determining whether the center point of the initial prediction box is on the road surface according to the road surface segmentation mask;

[0194] If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

[0195] In a possible implementation, the processing module 1001 is further configured to:

[0196] Performing segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features corresponding to the multi-layer feature map respectively;

[0197] The prediction module 1002 is further used for:

[0198] Prediction processing is performed according to the multiple two-dimensional instance features and the multiple obstacle features to obtain a road obstacle target in the input image.

[0199] In a possible implementation, the prediction module 1002 is further configured to:

[0200] Performing prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level;

[0201] Determining whether the center point of the initial prediction frame falls within the obstacle according to the multiple obstacle features;

[0202] If the center point of the initial prediction box falls within the obstacle, updating the first confidence level to a second confidence level, where the second confidence level is greater than the first confidence level;

[0203] If the second confidence level is greater than a preset value, the road obstacle target in the input image is determined according to the initial prediction frame.

[0204] In a possible implementation, the prediction module 1002 is further configured to:

[0205] The initial prediction box is used as the road obstacle target in the input image, or,

[0206] Determining whether the center point of the initial prediction box is on the road surface according to a road surface segmentation mask, wherein the road surface segmentation mask is obtained by processing the multi-layer feature map and the plurality of two-dimensional instance features;

[0207] If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

[0208] In a possible implementation, the processing module 1001 includes a trunk network and a main detection branch network, and the main detection branch network includes a neck network and a head network;

[0209] The feature extraction process of the input image to obtain the multi-layer feature map is implemented by the backbone network.

[0210] The neck network is used to fuse each layer of the feature map in the multi-layer feature map with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map respectively;

[0211] The road obstacle target in the input image is obtained by performing prediction processing according to the multiple two-dimensional instance features through the head network.

[0212] In a possible implementation, the backbone network and the neck network are trained in the following manner:

[0213] For the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing to obtain multiple sample feature maps of the sample image, where k is an integer not less than 1;

[0214] The multiple sample feature maps are input into the neck network N k Performing fusion processing in order to obtain a plurality of two-dimensional instance features respectively corresponding to the plurality of sample feature maps;

[0215] Obtaining a predicted value of a road segmentation mask according to the plurality of two-dimensional instance features and the plurality of sample feature maps;

[0216] A loss value is calculated according to the marked value and the predicted value, and a gradient is calculated according to the loss value;

[0217] Adjust the initial backbone network Z according to the gradient k and the initial neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

[0218] For the introduction of the above modules, please refer to the description of the above embodiments, which will not be repeated here.

[0219] It should be understood that the division of each module in each of the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the target detection device can be implemented in the form of a processor calling software; for example, the target detection device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each module of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, so as to realize the functions of some or all of the above units. All modules of the above devices can be realized in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the rest by hardware circuits.

[0220] Reference Fig.11 As shown in FIG. 1 , it is a schematic diagram of the hardware structure of another target detection device provided in an embodiment of the present application. Fig.11 The target detection device 1100 shown (the device 1100 may be a computer device) includes a memory 1101, a processor 1102, a communication interface 1103 and a bus 1104. The memory 1101, the processor 1102 and the communication interface 1103 are connected to each other through the bus 1104.

[0221] The memory 1101 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0222] The memory 1101 can store programs. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the target detection method of the embodiment of the present application.

[0223] The processor 1102 is a circuit with signal processing capability. In one implementation, the processor 1102 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor 1102 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 1102 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 1102 is used to execute relevant programs to implement the functions that need to be performed by the units in the target detection device of the embodiment of the present application, or to execute the target detection method of the method embodiment of the present application.

[0224] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0225] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0226] The communication interface 1103 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1100 and other devices or a communication network. For example, data can be acquired through the communication interface 1103 .

[0227] The bus 1104 may include a path for transmitting information between various components of the device 1100 (eg, the memory 1101 , the processor 1102 , and the communication interface 1103 ).

[0228] It should be noted that although Fig.11 The device 1100 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 1100 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 1100 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 1100 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig.11 All devices shown in .

[0229] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0230] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0231] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0232] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0235] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A target detection method, characterized in that: include: Performing feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map contains road feature information, and the multi-layer feature map has different downsampling rates; Each layer of the feature map in the multi-layer feature map is fused with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map; Performing prediction processing according to the multiple two-dimensional instance features to obtain a road obstacle target in the input image; The method further comprises: Performing segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features respectively corresponding to the multi-layer feature map; The step of performing prediction processing according to the plurality of two-dimensional instance features to obtain a road obstacle target in the input image includes: Prediction processing is performed according to the multiple two-dimensional instance features and the multiple obstacle features to obtain a road obstacle target in the input image.

2. The method according to claim 1, characterized in that The method further comprises: Processing the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask; The step of performing prediction processing according to the plurality of two-dimensional instance features to obtain a road obstacle target in the input image includes: Prediction processing is performed according to the multiple two-dimensional instance features and the road surface segmentation mask to obtain the road surface obstacle target in the input image.

3. The method according to claim 2, characterized in that The step of processing the multi-layer feature map and the plurality of two-dimensional instance features to obtain a road segmentation mask includes: According to the multi-layer feature map and the multiple two-dimensional instance features, a plurality of road surface features respectively corresponding to the multi-layer feature map are obtained; Wherein, for the i-th layer feature map, a fusion process is performed according to the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain a fused feature; The fused features are fused with the two-dimensional instance features corresponding to the i-th layer feature map to obtain the road surface features corresponding to the i-th layer feature map, wherein the plurality of road surface features corresponding to the multiple layers of feature maps respectively include the road surface features corresponding to the i-th layer feature map, i is an integer not less than 1, and when i is 1, the road surface features corresponding to the 0-th layer feature map do not exist; The road surface segmentation mask is obtained according to a plurality of road surface features respectively corresponding to the multi-layer feature maps.

4. The method according to claim 2, characterized in that: The step of performing prediction processing according to the plurality of two-dimensional instance features and the road segmentation mask to obtain a road obstacle target in the input image includes: Perform prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box; Determining whether the center point of the initial prediction frame is on the road surface according to the road surface segmentation mask; If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

5. The method according to claim 1, characterized in that The performing prediction processing according to the multiple two-dimensional instance features and the multiple obstacle features to obtain the road obstacle target in the input image includes: Performing prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level; Determining whether the center point of the initial prediction frame falls within the obstacle according to the multiple obstacle features; If the center point of the initial prediction box falls within the obstacle, updating the first confidence level to a second confidence level, where the second confidence level is greater than the first confidence level; If the second confidence level is greater than a preset value, the road obstacle target in the input image is determined according to the initial prediction frame.

6. The method according to claim 5, characterized in that The determining the road obstacle target in the input image according to the initial prediction frame includes: The initial prediction box is used as the road obstacle target in the input image, or, Determining whether the center point of the initial prediction frame is on the road surface according to a road surface segmentation mask, wherein the road surface segmentation mask is obtained by processing the multi-layer feature map and the plurality of two-dimensional instance features; If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

7. The method according to any one of claims 1 to 6, characterized in that: The prediction model includes a trunk network and a main detection branch network, wherein the main detection branch network includes a neck network and a head network; The feature extraction process of the input image to obtain the multi-layer feature map is implemented by the backbone network. The neck network is used to fuse each layer of the feature map in the multi-layer feature map with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map respectively; The road obstacle target in the input image is obtained by performing prediction processing according to the multiple two-dimensional instance features through the head network.

8. The method according to claim 7, characterized in that The backbone network and the neck network are trained in the following manner: For the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing to obtain a multi-layer sample feature map of the sample image, where k is an integer not less than 1; The multi-layer sample feature map is input into the neck network N k Performing fusion processing in order to obtain a plurality of two-dimensional instance features respectively corresponding to the multi-layer sample feature maps; Obtaining a predicted value of a road segmentation mask according to the plurality of two-dimensional instance features and the multi-layer sample feature map; A loss value is calculated according to the marked value and the predicted value, and a gradient is calculated according to the loss value; Adjust the backbone network Z according to the gradient k and the neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

9. A target detection device, characterized in that: include: A processing module, used for performing feature extraction processing on the input image to obtain a multi-layer feature map, wherein the feature map contains road feature information, and the downsampling rates of the multi-layer feature map are different; The processing module is further used to fuse each layer of the feature map in the multi-layer feature map with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map; A prediction module, configured to perform prediction processing according to the plurality of two-dimensional instance features to obtain a road obstacle target in the input image; The processing module is further used for: Performing segmentation mask prediction processing on the multi-layer feature map to obtain a plurality of obstacle features respectively corresponding to the multi-layer feature map; The prediction module is further used for: Prediction processing is performed according to the multiple two-dimensional instance features and the multiple obstacle features to obtain a road obstacle target in the input image.

10. The device according to claim 9, characterized in that The processing module is further used for: Processing the multi-layer feature map and the multiple two-dimensional instance features to obtain a road segmentation mask; The prediction module is further used for: Prediction processing is performed according to the multiple two-dimensional instance features and the road surface segmentation mask to obtain the road surface obstacle target in the input image.

11. The device according to claim 10, characterized in that The processing module is further used for: Processing is performed according to the multi-layer feature map and the multiple two-dimensional instance features to obtain multiple road surface features respectively corresponding to the multi-layer feature map; Wherein, for the i-th layer feature map, a fusion process is performed according to the road surface features corresponding to the i-th layer feature map and the i-1-th layer feature map to obtain a fused feature; The fused features are fused with the two-dimensional instance features corresponding to the i-th layer feature map to obtain the road surface features corresponding to the i-th layer feature map, wherein the plurality of road surface features corresponding to the multiple layers of feature maps respectively include the road surface features corresponding to the i-th layer feature map, i is an integer not less than 1, and when i is 1, the road surface features corresponding to the 0-th layer feature map do not exist; The road surface segmentation mask is obtained according to a plurality of road surface features respectively corresponding to the multi-layer feature maps.

12. The device according to claim 10, characterized in that The prediction module is further used for: Perform prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box; Determining whether the center point of the initial prediction frame is on the road surface according to the road surface segmentation mask; If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

13. The device according to claim 9, characterized in that The prediction module is further used for: Performing prediction processing according to the multiple two-dimensional instance features to obtain an initial prediction box and a first confidence level; Determining whether the center point of the initial prediction frame falls within the obstacle according to the obstacle feature; If the center point of the initial prediction box falls within the obstacle, updating the first confidence level to a second confidence level, where the second confidence level is greater than the first confidence level; If the second confidence level is greater than a preset value, the road obstacle target in the input image is determined according to the initial prediction frame.

14. The device according to claim 13, characterized in that The prediction module is further used for: The initial prediction box is used as the road obstacle target in the input image, or, Determining whether the center point of the initial prediction frame is on the road surface according to a road surface segmentation mask, wherein the road surface segmentation mask is obtained by processing the multi-layer feature map and the plurality of two-dimensional instance features; If the center point of the initial prediction box is on the road surface, the initial prediction box is used as the road obstacle target in the input image.

15. The device according to any one of claims 9 to 14, characterized in that The processing module includes a trunk network and a main detection branch network, and the main detection branch network includes a neck network and a head network; The feature extraction process of the input image to obtain the multi-layer feature map is implemented by the backbone network. The neck network is used to fuse each layer of the feature map in the multi-layer feature map with the feature map in the previous layer to obtain a plurality of two-dimensional instance features corresponding to the multi-layer feature map respectively; The road obstacle target in the input image is obtained by performing prediction processing according to the multiple two-dimensional instance features through the head network.

16. The device according to claim 15, characterized in that The backbone network and the neck network are trained in the following manner: For the kth training, the sample image is input into the backbone network Z k Perform feature extraction processing to obtain a multi-layer sample feature map of the sample image, where k is an integer not less than 1; The multi-layer sample feature map is input into the neck network N k Performing fusion processing in order to obtain a plurality of two-dimensional instance features respectively corresponding to the multi-layer sample feature maps; Obtaining a predicted value of a road segmentation mask according to the plurality of two-dimensional instance features and the multi-layer sample feature map; A loss value is calculated according to the marked value and the predicted value, and a gradient is calculated according to the loss value; Adjust the backbone network Z according to the gradient k and the neck network N k Parameters, let k = k + 1, and repeat the above steps until k reaches the preset number of times, the backbone network Z k As the backbone network, the neck network N k As the neck network.

17. A target detection device, characterized in that: It comprises a processor and a communication interface, wherein the communication interface is used to receive and / or send data, and / or the communication interface is used to provide output and / or output for the processor, and the processor is used to call computer instructions to implement the method described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

Images