Target detection method and device for driving assistance, computer device and medium

Abstract

The application relates to a target detection method for driving assistance, comprising the following steps: acquiring a driving scene video, and acquiring a driving scene image at each moment of the driving scene video; determining a plurality of mutually non-overlapping basic candidate boxes on the driving scene image; determining a focus area of the driving scene image; determining a plurality of expansion candidate boxes with different scales near each basic candidate box in the focus area; and inputting the basic candidate boxes and the expansion candidate boxes into a target detection model to detect a target affecting driving behavior, so as to obtain a target detection result of the driving scene image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to a target detection method and device for driving assistance, a computer device and a medium. BACKGROUND

[0002] In the field of intelligent driving, the concept of vehicle-road cooperation is widely concerned. Target detection is the basis of vehicle-road cooperation, and various road events and traffic estimation are highly dependent on the results of target detection. Generally, multiple cameras are used to capture the driving scene, and target detection is performed on the captured video to achieve vehicle-road cooperation. The target detection model in vehicle-road cooperation needs to detect the target position and state that needs to be concerned in driving within a certain range (for example, 0 to 400 meters).

[0003] In vehicle-road cooperation, the position and state of the target within a certain distance need to be informed to the vehicle, so the target detection in vehicle-road cooperation pays great attention to both the far-end and near-end targets. Since the targets located at the far-end in the camera field of view are small in scale and densely distributed, the detection accuracy of the target detection model is required to be high. SUMMARY

[0004] The present application provides a target detection method and device for driving assistance, a computer device and a medium.

[0005] The first aspect of the present application relates to a target detection method for driving assistance, comprising: acquiring a driving scene video and acquiring a driving scene image of each time of the driving scene video; determining a plurality of mutually non-overlapping basic candidate boxes on the driving scene image; determining a focus area of the driving scene image; determining a plurality of expansion candidate boxes with different scales near each basic candidate box in the focus area; and inputting the basic candidate box and the expansion candidate box into a target detection model to detect a target affecting driving behavior, to obtain a target detection result of the driving scene image.

[0006] In an embodiment, the target detection method for driving assistance according to the present application further comprises: predicting a focus area of the next time based on the target detection result of the driving scene image of the current time, and updating the focus area using the predicted focus area of the next time.

[0007] In an embodiment, determining the focus area of the driving scene image comprises: acquiring a pre-specified area as the focus area.

[0008] In an embodiment, determining the focus region of the driving scene image comprises: determining a small target dense area in the driving scene image as the focus region, wherein the small target dense area is an area in which a size of the target in the driving scene image is less than a predetermined size threshold or a density is greater than a predetermined density threshold.

[0009] In an embodiment, determining the small target dense area in the driving scene image as the focus region comprises: inputting a base candidate box of the driving scene image at an initial time of the driving scene video into the target detection model to detect the target, to obtain a target detection result at the initial time; and determining the small target dense area based on the target detection result at the initial time as the focus region.

[0010] In an embodiment, determining a plurality of mutually non-overlapping base candidate boxes on the driving scene image comprises: determining square base candidate boxes, a side length of the base candidate box being S, and S being a multiple of 2 of a down-sampling multiple of the target detection model x wherein x is an integer; determining a plurality of expansion candidate boxes of different sizes near each of the base candidate boxes in the focus region comprises: determining a plurality of first expansion candidate boxes near each of the base candidate boxes, wherein the first expansion candidate boxes comprise a plurality of rectangles with side lengths being multiples of S of positive integers, and centers of the plurality of first expansion candidate boxes coincide with a center of the base candidate box, and determining a plurality of second expansion candidate boxes by translating the plurality of first expansion candidate boxes together with the base candidate box to a plurality of directions on the driving scene image.

[0011] In an embodiment, before inputting the base candidate box and the expansion candidate box into the target detection model to detect the target affecting driving behavior, the method for driving assistance according to the present application further comprises: obtaining a sample image; determining a plurality of mutually non-overlapping base candidate boxes on the sample image, and determining a plurality of expansion candidate boxes of different sizes near each of the base candidate boxes; randomly retaining a part of the base candidate boxes and the expansion candidate boxes to obtain a plurality of retained candidate boxes; and inputting the retained candidate boxes into the target detection model for training.

[0012] In an embodiment, determining a plurality of mutually non-overlapping base candidate boxes on the sample image, and determining a plurality of expansion candidate boxes of different sizes near each of the base candidate boxes comprises: determining square base candidate boxes, a side length of the base candidate box being S, and S being a multiple of 2 of a down-sampling multiple of the target detection model xwherein x is an integer; determining a plurality of first extended candidate boxes around each of the base candidate boxes, wherein the first extended candidate boxes comprise a plurality of rectangles with a side length of S multiplied by a positive integer, and the centers of the first extended candidate boxes overlap with the center of the base candidate boxes, and determining a plurality of second extended candidate boxes by translating the first extended candidate boxes and the base candidate boxes to a plurality of directions on the sample image.

[0013] In an embodiment, a part of the base candidate boxes and the extended candidate boxes are randomly reserved to obtain a plurality of reserved candidate boxes, including one or more of the following steps: randomly selecting one or more regions in the sample image as reserved regions, and reserving only the candidate boxes located in the reserved regions as the reserved candidate boxes; randomly reserving a certain proportion of the candidate boxes in all the base candidate boxes and the extended candidate boxes as the reserved candidate boxes; and selecting the candidate boxes of a certain scale in the base candidate boxes and the extended candidate boxes as the reserved candidate boxes; or selecting the candidate boxes located at a certain position relative to the base candidate boxes as the reserved candidate boxes.

[0014] In an embodiment, the target detection method for driving assistance according to the present application further comprises: increasing the number of convolutional layers of a feature extraction network in the target detection model.

[0015] According to the target detection method for driving assistance according to the present application, for the attention region which is the region that needs to be focused in the driving scene, a larger number of candidate boxes with more scales are laid out, which is beneficial to improve the accuracy of target detection for the attention region; at the same time, it avoids increasing too many candidate boxes for the whole region of the image to be detected, and thus reduces the calculation amount of the target detection model, thereby improving the target detection speed. On the other hand, in the process of training the target detection model, in order to avoid laying too many candidate boxes on the sample image, which leads to introducing too many negative samples in the training process and thus leads to the decline of the detection accuracy of the model, a part of the candidate boxes are randomly discarded and only a part of them are reserved for training the target detection model, which can improve the detection stability of the target detection model while ensuring the detection accuracy.

[0016] According to a second aspect of the present application, a target detection device for driving assistance is provided, comprising: an acquisition module configured to acquire a driving assistance video of a target to be detected, and acquire a to-be-detected image of each time of the driving assistance video; a determination module configured to determine a plurality of mutually non-overlapping basic candidate boxes on the to-be-detected image, and determine a region of interest of the to-be-detected image, in which a plurality of extended candidate boxes with different scales are determined near each of the basic candidate boxes; and a detection module configured to input the basic candidate boxes and the extended candidate boxes into a target detection model for target detection, to obtain a target detection result of the to-be-detected image.

[0017] According to a third aspect of the present application, a computer device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the target detection method for driving assistance according to the first aspect of the present application when executing the computer program.

[0018] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the target detection method for driving assistance according to the first aspect of the present application.

[0019] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of the target detection method for driving assistance according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of determining a basic candidate box according to an embodiment of the present application;

[0022] Figure 3 is a flowchart of the target detection method for driving assistance according to an embodiment of the present application;

[0023] Figure 4A and Figure 4B is a schematic diagram of calculating an intersection over union according to an embodiment of the present application;

[0024] Figure 5 is a flowchart of the target detection method for driving assistance according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of determining a basic candidate box according to an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of determining a basic candidate box according to an embodiment of the present application;

[0027] Figure 8 is a flowchart of a target detection method for driving assistance according to an embodiment of the present application;

[0028] Figure 9 is a schematic diagram of a target detection device for driving assistance according to an embodiment of the present application;

[0029] Figure 10 is a schematic diagram of a target detection device for driving assistance according to an embodiment of the present application;

[0030] Figure 11 is a schematic diagram of a target detection device for driving assistance according to an embodiment of the present application;

[0031] Figure 12 is a schematic diagram of a target detection device for driving assistance according to an embodiment of the present application;

[0032] Figure 13 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the application of cooperative vehicle infrastructure system, a vehicle is expected to detect the position and state of targets within a certain range in its driving scene. In the present application, the target to be detected is, for example, an object in the driving scene that needs to be paid attention to, a target that may affect the driving behavior, etc., such as the position and movement state of nearby objects of the vehicle (such as vehicles, pedestrians, etc.), the position and state of a red light ahead, etc. The position and state of these targets will affect the driving behavior and decision of the vehicle. Therefore, a target detection model is usually used to detect the driving scene video taken. The accuracy and stability of the target detection result are particularly important.

[0034] The target detection model is a neural network model based on deep learning, which is used to detect the position and state of a target in an input video or image. In the process of detecting the target to be detected by using the target detection model, a plurality of candidate boxes are laid on the input image, and the target detection model judges the position and state of the target by judging whether the IOU (Intersection over Union) between each candidate box and the real box is greater than a predetermined threshold. Therefore, laying more candidate boxes on the image to be detected helps to obtain more output boxes with IOU greater than the threshold, so as to obtain more stable output results. However, if too many candidate boxes are laid on the image to be detected, the calculation amount of the model will increase, which reduces the processing speed, which will seriously affect the real-time detection of the driving scene, which is very disadvantageous in the application of cooperative vehicle infrastructure system.

[0035] Referring to Figure 1The target detection method for driving assistance according to the present application comprises the following steps S110-S190.

[0036] S110: Obtain a driving scene video and obtain a driving scene image of each time of the driving scene video.

[0037] The driving scene video can be a video of a scene outside a vehicle captured by a camera mounted on the vehicle in real time, or a road video captured by a camera movably or fixedly arranged on a road in real time. The driving scene video is a video to be subjected to target detection, and more specifically, a video to be detected for a target affecting driving behavior. The driving scene image of each time of the driving scene video is, for example, an image of each frame or an image obtained every several frames in the driving scene video. The driving scene image is, for example, an image obtained from the driving scene video in chronological order. The driving scene image, for example, includes a road and objects on the road in a driving scene of a vehicle.

[0038] S130: Determine a plurality of mutually non-overlapping basic candidate boxes on the driving scene image.

[0039] Further referring to Figure 2 , on the driving scene image 200, 8x5 mutually non-overlapping basic candidate boxes 201 are laid out. In Figure 2 , the plurality of basic candidate boxes 201 collectively completely cover the entire driving scene image 200. It should be understood that in some other embodiments, the plurality of basic candidate boxes 201 can only cover a part of the driving scene image 200.

[0040] The size or number of the basic candidate boxes 201 can be changed as needed. In some embodiments, the size and number of the candidate boxes are determined according to the size of a sample image. For example, square basic candidate boxes 201 are determined, and the side length of the basic candidate boxes 201 is S (e.g., S pixels). In Figure 2 the example shown, the size of the driving scene image 200 is MxN pixels, and basic candidate boxes will be determined, where S is a common divisor of M and N. Exemplarily, S is an integer power of 2 times of a down-sampling multiple of a target detection model used for target detection of the driving scene image, i.e., 2 x times, where x is an integer. Here, the integer can include positive integers, negative integers, and 0. For example, the integer power of 2 is, for example, 0.25, 0.5, 1, or 2, etc. For example, S is 0.25 times, 0.5 times, 1 times, 2 times, etc. of a down-sampling multiple of the target detection model.

[0041] S150: Determine a region of interest of the driving scene image.

[0042] The region of interest is a region in the driving scene image that needs to be focused on for target detection to detect targets that affect driving behavior. Illustratively, the region of interest is a region in the vehicle-road system that needs to be focused on in the driving scene and thus needs to be detected using higher precision. In other words, for the region of interest, the algorithm of the target detection model will need higher precision, and the target detection model will allocate more computational resources to the region of interest. In an embodiment, the area of the region of interest should be smaller than the overall area of the driving scene image, i.e., the region of interest is a part of the driving scene image.

[0043] In one embodiment, determining the region of interest of the driving scene image comprises obtaining a pre-specified region as the region of interest. For example, the region of interest is a region manually specified by a user. In this case, the driving scene video, and correspondingly the driving scene image, can be displayed on a display screen, e.g., a touch display screen, and the user can manually select a polygon on the display screen to specify the region inside the selected polygon as the region of interest. In this case, the user can select the region of interest according to his own needs. For example, in a driving scene video taken from an intersection, the content that needs to be focused on for the driving scene is usually the state of the traffic light at the intersection. Therefore, the user can manually select the region of the traffic light in the driving scene video as the region of interest.

[0044] In another embodiment, determining the region of interest of the driving scene image comprises determining a small target dense region in the driving scene image as the region of interest. The small target dense region is a region in the driving scene image where the scale of the targets is smaller than a predetermined scale threshold or the density of the targets is greater than a predetermined density threshold.

[0045] In the driving scene image, due to the small scale of the targets located at the far end of the field of view, if ordinary computational resources and precision are used for detection, the detection result will be very unstable, and the detection result of the small scale targets at the far end of the field of view will be very inaccurate. Small scale can be understood as the target to be detected covering a small area in the image to be detected. Similarly, if the density of the targets in the field of view of the driving scene image is large, in other words, the number of targets in a unit area is greater than a predetermined density threshold, i.e., the targets are densely distributed in a certain region, using ordinary computational resources and precision for detection will also result in a very unstable detection result, and the detection result of the densely distributed targets will be very inaccurate. Therefore, it is necessary to determine the small target dense region as the region of interest and use higher precision algorithms for detection. Illustratively, in the driving scene image, according to the imaging characteristics of the camera, the region of interest is usually the region at the far end of the field of view.

[0046] Further reference is made to Figure 3In some embodiments, determining the small target dense area in the driving scene image as the attention region comprises steps S310-S330.

[0047] S310: input the base candidate box of the driving scene image at the initial time of the driving scene video into the target detection model to detect the target, and obtain the target detection result at the initial time.

[0048] It should be understood that a driving scene video includes driving scene images at multiple time points, and the driving scene image at the initial time is the image at the starting time (e.g., the first frame or the first few frames in the video) in a driving scene video. The base candidate box is selected for the driving scene image at the initial time using the method as described above, and the selected base candidate box is input into the target detection model for target detection. The target detection result obtained at this time is the detection result for the driving scene image at the initial time.

[0049] S330: determine the small target dense area based on the target detection result at the initial time as the attention region.

[0050] Since fewer candidate boxes are applied to the initial image and each base candidate box does not overlap with each other, the detection result obtained for the image may not be very accurate, but the approximate size and position of the target can be known from the detection result. Based on the result of target detection on the driving scene image at the initial time, it can be determined which regions in the driving scene image have targets with a size less than a predetermined size threshold or a target density greater than a predetermined density threshold, thereby determining the attention region in the image. At this time, the attention region determined can be understood as the initial attention region in the process of target detection on the driving scene video.

[0051] Referring back to Figure 1 The target detection method for driving assistance according to the present application further comprises S170: determining a plurality of expansion candidate boxes with different sizes near each of the base candidate boxes in the attention region.

[0052] In the process of detection inference using the target detection model, the candidate box is scaled to obtain the real target region in combination with the offset of the target detection model. In detection, IOU is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. IOU calculates the overlap rate of the "predicted bounding box" and the "real bounding box". In actual calculation, IOU is the result of dividing the overlapping part of the two regions by the set part of the two regions. Using IOU can be used to judge whether the prediction result is correct or not. In practice, the commonly used IOU threshold is 0.5. If the IOU is greater than 0.5, it is considered that the box is a correct box (positive sample), otherwise it is considered to be an incorrect box (negative sample).

[0053] In the process of detection inference using the target detection model, the movement and scaling of the target to be detected may touch the boundary of the positive sample and the negative sample, and thus be classified as a negative sample. Therefore, if the number and density of the candidate boxes in the image to be detected are increased, the number of correct boxes obtained will be increased, and thus the detection result will be more stable. However, if more candidate boxes are laid for all regions of the driving scene image, a large number of candidate boxes will be increased, the calculation amount of the model will be increased, and the processing speed will be reduced. On the other hand, the large number of candidate boxes will also increase the number of error boxes output at the same time, thereby affecting the detection accuracy. This will seriously affect the detection real-time performance and accuracy of the driving scene, which is very disadvantageous in the application of vehicle-road cooperation.

[0054] Therefore, according to the scheme of the embodiments of the present application, only the extended candidate boxes are laid in the region of interest, and only the basic candidate boxes are used for target detection for the regions outside the region of interest in the driving scene image, which helps to reduce the calculation pressure of the algorithm of the target detection model, thereby improving the calculation speed and output accuracy. It should be understood that in the embodiments of the present application, the extended candidate boxes are only determined in the region of interest, and no extended candidate boxes are laid for the regions outside the region of interest in the driving scene image.

[0055] In specific applications, the coverage area of the extended candidate box is not completely coincident with the coverage area of the basic candidate box, that is, the area covered by the extended candidate box may overlap with the coverage area of the basic candidate box, but the coverage areas of the two are not completely the same. In addition, the scales of the plurality of extended candidate boxes are different, and accordingly, the plurality of extended candidate boxes can cover different areas of the driving scene image, which is beneficial to the detection of targets of different scales. It should be understood that the determination of a plurality of extended candidate boxes near each basic candidate box means that there is an overlapping part between the extended candidate box and the basic candidate box, for example, each extended candidate box has an overlapping part with the basic candidate box, and the plurality of extended candidate boxes can have an overlapping part with each other. Preferably, the area of each extended candidate box is not less than the area of the basic candidate box.

[0056] In some embodiments, the determination of a plurality of candidate boxes in the region of interest means that the entire area of each extended candidate box is located within the region of interest, and in other embodiments, it can also mean that a part of the area of each candidate box is located within the region of interest.

[0057] In specific examples, since the probability of targets of different scales touching the boundary of the positive sample and the negative sample is not the same, preferably, the small target dense area in the driving scene image is set as the region of interest. Specifically, referring to Figure 4A , A is a real bounding box, B and C are respectively a candidate box before movement and a candidate box after movement, and the target to be detected is a target with a smaller scale in the image. Before the target moves, and after the target moves, Therefore, for a small-scale target, slight movement of the target to be detected will result in a great change in the IOU. Conversely, referring to Figure 4B , A is a real bounding box, B and C are respectively a candidate box before movement and a candidate box after movement, and the target to be detected is a large-scale target in the image. Before the target moves, While after the target moves, Therefore, for a large-scale target, slight movement of the target to be detected will not result in a great change in the IOU.

[0058] Based on this, only for the region in the driving scene image where small targets are distributed, more candidate boxes (i.e., extended candidate boxes) are laid out, and for other regions, only basic candidate boxes are laid out. In this way, the target detection method for driving assistance according to the present application can improve the detection accuracy of the remote small target dense area without increasing the time consumption, in other words, can reduce the number of overall candidate boxes while ensuring the detection accuracy, and improve the detection efficiency.

[0059] Similarly, only for the region in the driving scene image where the target to be detected is dense, extended candidate boxes are laid out, and for other regions, only basic candidate boxes are laid out. Preferably, only for the region in the driving scene image where the target to be detected is smaller than a predetermined scale threshold and the density is greater than a predetermined density threshold, extended candidate boxes are laid out.

[0060] Continuing to refer to Figure 1 The target detection method for driving assistance according to the present application further comprises S190: inputting the basic candidate boxes and the extended candidate boxes into a target detection model to detect targets affecting driving behavior, to obtain a target detection result of the driving scene image.

[0061] It should be understood that the target detection result of each moment of the driving scene video is obtained, and each moment of the driving scene image can be fed back to the vehicle or the driver of the vehicle in real time. In this way, the vehicle or the driver of the vehicle makes a decision on driving behavior according to the target detection result. Accordingly, the target detection result of each moment of the driving scene image can be obtained, and thus the target detection result of the driving scene video is obtained.

[0062] In some embodiments, the target detection method for driving assistance according to the present application further comprises: predicting a focus region of the next moment based on the target detection result of the driving scene image of the current moment, and updating the focus region using the predicted focus region of the next moment.

[0063] As described in the preceding steps, the initial region of interest (ROI) can be obtained by inputting the basic candidate bounding boxes of the driving scene image at the initial moment of the driving scene video into the object detection model. Correspondingly, the ROI for each subsequent moment of the driving scene video can be obtained based on the object detection results of the previous moment, because objects in video object detection are continuously changing without abrupt changes. Based on the motion patterns of the objects, the algorithm can predict the target scale and density distribution at the next moment based on the target detection results at the current moment. Therefore, the ROI for subsequent moments can be continuously determined based on the initial ROI, and the ROI can be updated in real time. Specifically, when the ROI is the aforementioned dense area of ​​small objects, the dense area of ​​small objects at the current moment is used to predict the dense area of ​​small objects at the next moment, and the ROI is updated using the predicted dense area of ​​small objects at the next moment. In this way, the accuracy of the algorithm in the object detection model for each region can be adjusted in real time, using a more computationally intensive and more accurate algorithm in dense areas of small objects, and a less computationally intensive and moderately accurate algorithm in areas where the distribution of the objects to be detected is sparse. The region of interest is dynamically adjusted during the target detection process in the driving scene video. That is, the region where expanded candidate boxes are laid out in the driving scene image is dynamically adjusted accordingly. The algorithm in the target detection model adjusts the region of interest based on the real-time target state (e.g., scale and density). Thus, the target detection method for driver assistance according to this application can achieve real-time, dynamic adjustment of the detection accuracy of each region in the driving scene video.

[0064] In some embodiments, see further. Figures 5 to 6 S170 determines multiple extended candidate boxes of different scales near each of the basic candidate boxes in the region of interest, specifically including the following steps S510-S530.

[0065] S510: Determine a plurality of first extended candidate boxes near each of the basic candidate boxes, wherein the first extended candidate boxes include a plurality of rectangles with side lengths that are positive integer multiples of S, and the centers of the plurality of first extended candidate boxes coincide with the centers of the basic candidate boxes.

[0066] In such Figure 6 In the example shown, the solid-line box in the center ( Figure 6 The shaded area in the image represents the base candidate boxes, and the remaining dashed boxes are the first expanded candidate boxes. Assume the side length of the base candidate box is S, and the first expanded candidate boxes include rectangles with side lengths of 2S and 4S respectively. Then, the side lengths of the three rectangles with side lengths of S, 2S, and 4S (one base candidate box and two first expanded candidate boxes) are stretched by 1:2 and 2:1 respectively, resulting in 9 candidate boxes (e.g., in...). Figure 6The example shows eight first expanded candidate boxes and one base candidate box. For example, stretching the side length of a rectangle with side length S by 1:2 and 2:1 will result in rectangles with side lengths of S×2S and 2S×S; similarly, stretching the side length of a rectangle with side length 2S by 1:2 and 2:1 will result in rectangles with side lengths of 2S×4S and 4S×2S, and stretching the side length of a rectangle with side length 4S by 1:2 and 2:1 will result in rectangles with side lengths of 4S×8S and 8S×4S. In this example, these rectangles with side lengths of 2S×2S, 4S×4S, S×2S, 2S×S, 2S×4S, 4S×2S, 4S×8S, and 8S×4S are all first expanded candidate boxes.

[0067] like Figure 6 As shown, the centers of the multiple first expanded candidate boxes coincide with the centers of the base candidate boxes. Preferably, the axes of symmetry of the multiple first expanded candidate boxes coincide with the axes of symmetry of the base candidate boxes. It should be understood that the number of first expanded candidate boxes can vary as needed and is not limited to the number described in the embodiments. Furthermore, the scale of the first expanded candidate boxes can also vary as needed, and their side lengths are not limited to 1, 2, 4, or 8 times the side length of the base candidate box, but can also be other multiples. It should also be understood that there are various different methods for determining the first expanded candidate boxes; that is, the position and scale of the first expanded candidate boxes can be changed as needed, as long as the first expanded candidate boxes are located near the base candidate boxes.

[0068] S530: By translating the plurality of first extended candidate boxes together with the base candidate boxes in multiple directions on the driving scene image, a plurality of second extended candidate boxes are determined.

[0069] For example, such as Figure 7 As shown, the side length of the basic candidate box is S. A basic candidate box and the multiple first extended candidate boxes corresponding to it, obtained as above, are grouped into a set of candidate boxes. This set of candidate boxes is then translated by a distance of 0.5S in two mutually perpendicular side length directions, and also translated diagonally. The distance is calculated, resulting in three sets of translated candidate boxes as the second expanded candidate boxes. Each set of candidate boxes has a center point O. The lines connecting the center points O of the four sets of candidate boxes obtained by the above translation will form a rectangle with a side length of 0.5S.

[0070] It should be understood that, Figure 7 In the example shown, to avoid overlapping candidate boxes in the schematic diagram affecting readability, the translation distance of the candidate boxes is not illustrated as the actual distance. In practice, preferably, each first expanded candidate box and each second expanded candidate box has a portion overlapping with the base candidate box.

[0071] As can be seen from the above, the expanded candidate boxes include a first expanded candidate box and a second expanded candidate box, that is, in Figure 7 In the examples shown, the rectangles without shadow fill are all expanded candidate boxes. Figure 7 The example shown contains 36 candidate boxes: one base candidate box and 35 expanded candidate boxes (8 first expanded candidate boxes and 27 second expanded candidate boxes). In other words, one base candidate box corresponds to 35 expanded candidate boxes. However, as... Figure 7 The embodiments shown are merely examples. The distance and direction of translation of a set of candidate boxes can be changed according to actual needs, and are not limited to the distance and direction described in the embodiments. Similarly, the number of translations of a set of candidate boxes (the corresponding number of second expanded candidate boxes) can also be changed according to actual needs, and are not limited to the number of translations described in the embodiments.

[0072] On the other hand, the target detection method for driving assistance according to this application further includes a process of training the target detection model before using the target detection model to detect targets in the driving scene video. Specifically, see... Figure 8 Before inputting the basic candidate box and the extended candidate box into the target detection model to detect targets that affect driving behavior, the method further includes the following steps S810-S870.

[0073] S810: Acquire sample image.

[0074] Sample images can be pre-prepared images of driving scenarios, such as road images captured by a camera.

[0075] S830: Determine multiple non-overlapping basic candidate boxes on the sample image, and determine multiple extended candidate boxes of different scales near each of the basic candidate boxes.

[0076] The method for determining the basic candidate box and the extended candidate box on the sample image can refer to the method for determining the basic candidate box and the extended candidate box on the driving scene image to be detected in the above embodiment, and will not be repeated here.

[0077] S850: Randomly retain a portion of the candidate boxes in the basic candidate box and the extended candidate box to obtain multiple retained candidate boxes.

[0078] During the training phase of the object detection model, candidate boxes determined on the sample image with an Interchange of Value (IOU) greater than a certain positive sample threshold will be classified as positive samples, and those less than a certain negative sample threshold will be classified as negative samples. Candidate boxes between the positive and negative sample thresholds can be discarded. For some adaptive positive and negative sample allocation algorithms, candidate boxes may need to meet more conditions to be classified as positive samples.

[0079] The increase of the density of the candidate boxes will make the number of positive samples meeting the condition larger, so that the probability of the target in the image to be detected touching the boundary of the positive and negative samples is reduced in the detection inference stage using the model. Therefore, the density and scale of the candidate boxes determined on the image directly determine the stability of the final detection result. However, too many candidate boxes will introduce more negative samples in the training stage of the target detection model, resulting in the decline of the detection accuracy and the calculation speed in the subsequent inference stage. Therefore, in the training stage of the target detection model, the accuracy and stability of the target detection model need to be considered.

[0080] Therefore, in order to make all the candidate boxes be fully trained and reduce the training difficulty, the candidate boxes on the sample image are randomly discarded, that is, a part of the candidate boxes in the basic candidate boxes and the extended candidate boxes are randomly reserved, and the rest of the candidate boxes are discarded. When the candidate boxes are reserved and discarded, the randomness of reservation and discarding should be focused on, so that the candidate boxes at various positions and various scales on the sample image are reserved in a certain number, thereby improving the effectiveness of training.

[0081] S870: inputting the reserved candidate boxes to the target detection model for training.

[0082] As described above, when training the target detection model, only a part of the candidate boxes on the sample image is used to train the model, which helps to lay more candidate boxes while ensuring the accuracy of the model, so as to improve the training efficiency. It should be understood that the discarded candidate boxes neither serve as positive samples nor negative samples, do not participate in training, and do not participate in the calculation of the loss function. Thus, the trained target detection model is obtained. In the actual detection inference process, the trained target detection model is used to detect the target in the driving scene image and the driving scene video.

[0083] In some embodiments, a part of the candidate boxes in the basic candidate boxes and the extended candidate boxes are randomly reserved to obtain a plurality of reserved candidate boxes, including one or more of the following steps (1)-(3):

[0084] (1) randomly selecting one or more regions in the sample image as reserved regions, and only reserving the candidate boxes located in the reserved regions as the reserved candidate boxes.

[0085] In other words, one or more discarded regions can be randomly selected on the sample image, and all the candidate boxes located in the one or more discarded regions are discarded. The retained regions and the discarded regions are mutually exclusive in the sample image, and the area of the retained regions and the sum of the areas of the discarded regions are the area of the sample image. The candidate boxes located in the retained regions are retained, and the candidate boxes located in the discarded regions are discarded. The candidate boxes located in the retained regions or the discarded regions can be understood as the candidate boxes whose entire areas are located in the regions, or can be understood as the candidate boxes whose only part of the areas are located in the regions. It should be understood that in the present application, the candidate boxes include the basic candidate boxes and the extended candidate boxes.

[0086] In an embodiment, the selection of the retained regions or the discarded regions is random. For example, in the process of multiple training and multiple selection of the retained regions or the discarded regions, the retained regions or the discarded regions selected each time are not the same.

[0087] (2) A certain proportion of the candidate boxes among all the candidate boxes in the basic candidate boxes and the extended candidate boxes are randomly retained as the retained candidate boxes.

[0088] For example, 20%-50% of all the candidate boxes can be randomly discarded, and 50%-80% of the candidate boxes can be correspondingly retained as the retained candidate boxes. For example, when there are 2000 candidate boxes (including the basic candidate boxes and the extended candidate boxes) on the sample image, 1000-1600 candidate boxes on the sample image can be randomly retained, and the remaining candidate boxes are discarded.

[0089] (3) The candidate boxes of a specific scale are selected from the basic candidate boxes and the extended candidate boxes as the retained candidate boxes; or the candidate boxes located at a specific position relative to the basic candidate boxes are selected as the retained candidate boxes.

[0090] For example, the side length of the basic candidate box is S, and S is 2 x times (2x times) of the down-sampling multiple of the target detection model, where x is an integer. The candidate boxes of a specific scale can be selected as the retained candidate boxes. For example, the candidate boxes with a side length of 4S can be selected as the retained candidate boxes, and the remaining candidate boxes are discarded. For example, the candidate boxes with a scale size of 4Sx4S, 2Sx4S, 4Sx2S, 4Sx8S and 8Sx4S are selected as the retained candidate boxes. Alternatively or additionally, the extended candidate boxes located at a specific position relative to the basic candidate boxes can be selected as the retained candidate boxes. For example, the candidate boxes with their centers located at a distance of 0.5S to the right of the center of the basic candidate box are selected as the retained candidate boxes. It can be understood that the selection of the scale and the position of the candidate boxes in the above examples is only exemplary, and can be changed as needed in practice.

[0091] The step of randomly reserving the bounding boxes includes one or more of the steps (1)-(3) described above, which can be using different steps of the steps (1)-(3) described above in multiple training respectively, or applying each of the steps (1)-(3) described above in each training.

[0092] In this way, it is avoided that too many bounding boxes are used in the training of the target detection model, while the training efficiency is ensured as much as possible and thus the accuracy of the target detection model is ensured.

[0093] In some embodiments, the driving assistance target detection method according to the present application further includes: increasing the number of convolution layers of the feature extraction network in the target detection model.

[0094] Since the number of bounding boxes input to the target detection model is increased, the processing amount of the target detection model is increased. In order to improve the generalization ability of the target detection model, it is beneficial to increase the feature width of the last two layers of the target detection model, i.e., correspondingly increase the number of convolution layers of the feature extraction network in the target detection model, and increase the number of output channels. In this way, it is beneficial to improve the stability of the output result in the detection inference process using the target detection model.

[0095] In the above embodiment of the target detection method for driving assistance, it should be understood that the order of the above steps is only exemplary and is intended for convenience of description. In actual application, the order of the above steps can be changed as long as there is no logical contradiction.

[0096] The target detection method for driving assistance according to the above embodiment lays more number and more diverse scales of bounding boxes on the attention region which is the region that needs to be focused on in the driving scene, which is beneficial to improve the accuracy of target detection on the attention region; at the same time, it avoids adding too many bounding boxes to the whole region of the image to be detected, and thus reduces the calculation amount of the target detection model, thereby improving the target detection speed. Specifically, for the small target dense area which needs to be focused on in the car-road cooperation application, a higher-precision algorithm is used for detection, and the small target dense area can be dynamically adjusted in the process of detecting the driving scene video.

[0097] On the other hand, in the process of training the target detection model, in order to avoid that laying too many bounding boxes on the sample image leads to too many negative samples being introduced in the training process and thus leads to the decline of the detection accuracy of the model, randomly discarding and reserving only a part of the bounding boxes for training the target detection model can improve the detection stability of the target detection model while ensuring the detection accuracy.

[0098] According to another aspect of the present application, as Figure 9As shown, a target detection apparatus 900 for driving assistance is provided, comprising an acquisition module 910, a determination module 930 and a detection module 950. The acquisition module 910 is configured to acquire a driving assistance video of a target to be detected, and acquire a to-be-detected image of each time instant of the driving assistance video. The determination module 930 is configured to determine a plurality of mutually non-overlapping basic candidate boxes on the to-be-detected image, and determine a region of interest of the to-be-detected image, in which a plurality of extended candidate boxes of different scales are determined near each of the basic candidate boxes. The detection module 950 is configured to input the basic candidate boxes and the extended candidate boxes into a target detection model for target detection, to obtain a target detection result of the to-be-detected image.

[0099] In an embodiment, as shown in Figure 10 The target detection apparatus 900 further comprises a prediction module 970 configured to predict a region of interest of a next time instant based on a target detection result of a driving scene image of a current time instant, and update the region of interest by using the predicted region of interest of the next time instant.

[0100] In an embodiment, the determination module 930 is further configured to acquire a pre-specified region as the region of interest.

[0101] In an embodiment, the determination module 930 is further configured to determine a small target dense area in the driving scene image as the region of interest, wherein the small target dense area is an area in which a scale of the target is smaller than a predetermined scale threshold or a density is greater than a predetermined density threshold in the driving scene image. Specifically, the determination module 930 is configured to input a basic candidate box of the driving scene image of an initial time instant of the driving scene video into the target detection model to detect the target, to obtain a target detection result of the initial time instant; and determine the small target dense area as the region of interest based on the target detection result of the initial time instant.

[0102] In an embodiment, the determination module 930 is further configured to: determine a square basic candidate box, a side length of the basic candidate box being S, and S being an integer multiple of 2 x of a down-sampling factor of the target detection model, wherein x is an integer; determine a plurality of first extended candidate boxes near each of the basic candidate boxes, wherein the first extended candidate boxes comprise a plurality of rectangles with side lengths being integer multiples of S, and centers of the plurality of first extended candidate boxes coincide with a center of the basic candidate box, and a plurality of second extended candidate boxes are determined by translating the plurality of first extended candidate boxes and the basic candidate box together in a plurality of directions on the driving scene image.

[0103] In an embodiment, as shown in Figure 11As shown, the target detection apparatus 900 further comprises a training module 940 configured to, before inputting the basic candidate boxes and the extended candidate boxes into the target detection model to detect the target affecting the driving behavior, acquire a sample image; determine a plurality of basic candidate boxes that do not overlap with each other on the sample image, and determine a plurality of extended candidate boxes with different scales near each basic candidate box; randomly reserve a part of the candidate boxes in the basic candidate boxes and the extended candidate boxes to obtain a plurality of reserved candidate boxes; and input the reserved candidate boxes into the target detection model for training.

[0104] In an embodiment, the training module 940 is further configured to implement one or more of the following steps: randomly select one or more regions in the sample image as reserved regions, and reserve only the candidate boxes located in the reserved regions as the reserved candidate boxes; randomly reserve a certain proportion of the candidate boxes in all the candidate boxes in the basic candidate boxes and the extended candidate boxes as the reserved candidate boxes; and select candidate boxes with specific scales in the basic candidate boxes and the extended candidate boxes as the reserved candidate boxes; or select candidate boxes located at specific positions relative to the basic candidate boxes as the reserved candidate boxes.

[0105] In an embodiment, as shown in Figure 12 The target detection apparatus 900 further comprises a network layer increasing module 990 configured to increase the number of convolutional layers of a feature extraction network in the target detection model.

[0106] The target detection apparatus for driving assistance of the present application corresponds to the target detection method for driving assistance of the present application, and the technical features and advantages described in the embodiments of the target detection method for driving assistance are applicable to the embodiments of the target detection apparatus for driving assistance, and hereby declared.

[0107] According to another aspect of the present application, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 13As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores the step system and the computer program. The internal memory provides an environment for the running of the step system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize the target detection method for driving assistance according to the above-mentioned embodiments. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0108] Those skilled in the art can understand that, Figure 13 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In an embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the steps in the above method embodiments.

[0110] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps in the above method embodiments.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0112] The technical features of the above-mentioned embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0113] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A target detection method for driving assistance, comprising: Acquire driving scene video, and acquire driving scene images at each moment of the driving scene video; Multiple non-overlapping basic candidate boxes are determined on the driving scene image; Determine the region of interest in the driving scene image; In the region of interest, multiple expanded candidate boxes of different scales are determined near each of the basic candidate boxes. as well as The basic candidate box and the extended candidate box are input into the target detection model to detect targets that affect driving behavior, and the target detection results of the driving scene image are obtained.

2. The method according to claim 1, further comprising: Based on the target detection results of the driving scene image at the current moment, the region of interest for the next moment is predicted, and the region of interest for the next moment is updated using the predicted region of interest for the next moment.

3. The method according to claim 1, wherein, Determining the region of interest in the driving scene image includes: Obtain a pre-specified region as the region of interest.

4. The method according to claim 1, wherein, Determining the region of interest in the driving scene image includes: The region of interest is determined as a dense area of ​​small targets in the driving scene image, wherein the dense area of ​​small targets is a region in the driving scene image where the scale of the targets is smaller than a predetermined scale threshold or the density is greater than a predetermined density threshold.

5. The method according to claim 4, wherein, Determining the densely populated area of ​​small targets in the driving scene image as the region of interest includes: The basic candidate bounding boxes of the driving scene image at the initial moment of the driving scene video are input into the target detection model to detect the target, thereby obtaining the target detection result at the initial moment; and The small target density region is determined based on the target detection results at the initial moment, and is taken as the region of interest.

6. The method according to any one of claims 1 to 5, wherein, Determining multiple non-overlapping basic candidate boxes on the driving scene image includes: Determine the basic candidate bounding box of the square, wherein the side length of the basic candidate bounding box is S, and S is 2 times the downsampling factor of the target detection model. x times, where x is an integer; Determining multiple expanded candidate boxes of different scales near each of the base candidate boxes within the region of interest includes: Multiple first expanded candidate boxes are determined near each of the base candidate boxes. Each first expanded candidate box comprises multiple rectangles with side lengths that are positive integer multiples of S, and the centers of the multiple first expanded candidate boxes coincide with the centers of the base candidate boxes. Multiple second extended candidate boxes are determined by translating the multiple first extended candidate boxes together with the base candidate boxes in multiple directions on the driving scene image.

7. The method according to any one of claims 1 to 5, further comprising, before inputting the basic candidate box and the extended candidate box into the target detection model to detect targets affecting driving behavior: Acquire sample images; Multiple non-overlapping basic candidate boxes are determined on the sample image, and multiple extended candidate boxes of different scales are determined near each of the basic candidate boxes; Randomly retain a portion of the candidate boxes in the basic candidate boxes and the expanded candidate boxes to obtain multiple retained candidate boxes; as well as The retained candidate boxes are input into the object detection model for training.

8. The method according to claim 7, wherein, Multiple non-overlapping basic candidate boxes are determined on the sample image, and multiple expanded candidate boxes of different scales are determined near each basic candidate box, including: Determine the basic candidate bounding box of the square, wherein the side length of the basic candidate bounding box is S, and S is 2 times the downsampling factor of the target detection model. x times, where x is an integer; Multiple first expanded candidate boxes are determined near each of the base candidate boxes, wherein each first expanded candidate box comprises multiple rectangles with side lengths that are positive integer multiples of S, and the centers of the multiple first expanded candidate boxes overlap with the centers of the base candidate boxes. Multiple second expanded candidate boxes are determined by translating the multiple first expanded candidate boxes together with the base candidate boxes in multiple directions on the sample image.

9. The method according to claim 7, wherein, Randomly retain a portion of the candidate boxes from the base candidate boxes and the expanded candidate boxes to obtain multiple retained candidate boxes, including one or more of the following steps: One or more regions in the sample image are randomly selected as retention regions, and only candidate boxes located in the retention regions are retained as the candidate boxes to be retained. A certain percentage of candidate boxes from all candidate boxes in the basic candidate box and the extended candidate box are randomly retained as the retained candidate boxes; as well as A candidate box of a specific scale is selected from the basic candidate box and the expanded candidate box as the retained candidate box; Alternatively, a candidate box located at a specific position relative to the base candidate box can be selected as the retained candidate box.

10. The method of claim 7, further comprising: Increase the number of convolutional layers in the feature extraction network of the target detection model.

11. A target detection device for driving assistance, comprising: The acquisition module is used to acquire the driving assistance video of the target to be detected, and to acquire the image to be detected at each moment of the driving assistance video; The determination module is used to determine multiple non-overlapping basic candidate boxes on the image to be detected, and to determine the region of interest of the image to be detected, and to determine multiple extended candidate boxes of different scales near each of the basic candidate boxes in the region of interest; as well as The detection module is used to input the basic candidate box and the extended candidate box into the target detection model to perform target detection and obtain the target detection result of the image to be detected.

12. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images