Method and device for detecting a target in a fisheye image

Abstract

The application discloses a fisheye image target detection method and device, the method comprising: inputting the fisheye image into a pre-trained target detection model; using a convolutional neural network module to identify the target object in the fisheye image; using a rotating box target detection module to rotate the detection frame to a predicted angle, obtaining a target detection frame, and outputting a first image; the first image is a fisheye image containing the target detection frame; using a self-attention algorithm module, based on the background area of the first image, enhancing the feature area of the first image, and outputting the enhanced first image as the detection result; the background area is an area outside the target detection frame in the first image, and the feature area is an area inside the target detection frame. The scheme of the application can effectively reduce the background information in the detection frame by rotating the detection frame to identify the target object in the fisheye image, thereby improving the target detection accuracy of the fisheye image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for target detection in fisheye images. Background Technology

[0002] Fisheye images are similar to compressing and distorting light over a wide angle range, squeezing the image into a relatively small space. Because of their wider field of view, fisheye images can be used for detection and early warning in autonomous driving. Specifically, target detection can be performed on fisheye images, and the location information of detected targets can be sent to the driving system, allowing the vehicle to be aware of the surrounding environment in a timely manner. These targets can include vehicles, pedestrians, bicycles, etc.

[0003] Currently, the most commonly used object detection method is the Convolutional Neural Networks (CNN) method. However, this method is suitable for detecting objects with horizontal bounding boxes in regular images. Since objects in fisheye images are presented at arbitrary angles, using horizontal bounding box methods would result in the inclusion of too much useless background information within the bounding box, leading to lower object detection accuracy in fisheye images. Summary of the Invention

[0004] This invention provides a target detection method and apparatus for fisheye images, which solves the problems of low target detection accuracy in fisheye images caused by low target-target fit and large amount of background information contained in the target box when using horizontal bounding boxes to identify targets in fisheye images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a target detection method for fisheye images, the method comprising:

[0007] Acquire fisheye images within a preset range of the road surface;

[0008] The fisheye image is input into a pre-trained target detection model; the target detection model includes a convolutional neural network module, a rotated bounding box target detection module, and a self-attention algorithm module;

[0009] The convolutional neural network module is used to identify target objects in the fisheye image, and the identified target objects are labeled using detection boxes;

[0010] The rotating box target detection module is used to rotate the detection box by a predicted angle to obtain the rotating detection box with the least noise information as the target detection box, and a first image is output; the first image is a fisheye image containing the target detection box;

[0011] Using the self-attention algorithm module, the feature region of the first image is enhanced based on the background region of the first image, and the enhanced first image is output as the detection result; the background region is the region in the first image located outside the target detection box, and the feature region is the region located inside the target detection box.

[0012] In one possible implementation, rotating the detection box by a predicted angle to obtain a rotated detection box containing the least noise information as the target detection box specifically includes:

[0013] Obtain the position of the center point and the position of the long side of the detection frame;

[0014] Using the center point as the origin of the coordinate system and the straight line passing through the origin and parallel to the long side as the x-axis, the detection box is rotated clockwise around the center point by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box; the predicted angle ∈ [0°, 180°); the predicted angle is the angle between the x-axis of the detection box and the x-axis of the rotated detection box when the x-axis of the detection box and the long side of the rotated detection box begin to coincide.

[0015] In one possible implementation, the object detection model is trained using the following method:

[0016] Acquire training samples, which include sample images and label information of target objects in the sample images. The label information includes the position label of the detection box of the target object, the angle label of the detection box, the category label of the target object, and the size label of the target object.

[0017] The sample image is input into the initial detection model, which includes an initial convolutional neural network module, an initial rotated box target detection module, and an initial self-attention algorithm module.

[0018] The initial convolutional neural network module is used to detect target objects in the sample image, and the target objects are labeled with detection boxes;

[0019] The detection box is rotated by a predicted angle using the initial rotated box target detection module to obtain the rotated detection box with the least noise information as the target detection box, and the first image is output.

[0020] The initial self-attention algorithm module is used to enhance the feature region of the first image based on the background region of the first image. The enhanced first image is output as the detection result, and the predicted position of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box are obtained.

[0021] Multiple sub-loss functions are used to determine the sub-loss values ​​between the predicted position, the predicted category, the predicted angle, the predicted size and the corresponding label information, and the target loss value for this training is determined based on the sub-loss values.

[0022] The parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module are modified according to the target loss value, and training continues using the training samples until the loss value between the predicted feature information and the label information is less than a preset threshold. Training ends, and the object detection model is obtained.

[0023] In one possible implementation, determining the sub-loss values ​​between the predicted position, the predicted category, the predicted angle, the predicted size, and the corresponding label information using multiple sub-loss functions specifically includes:

[0024] The loss value between the predicted position of the detection box and the position label of the detection box is determined by using a preset detection box sub-loss function, and the first sub-loss value is obtained.

[0025] The loss value between the predicted category of the target object and the category label of the target object is determined by using a preset target sub-loss function, and a second sub-loss value is obtained;

[0026] The loss value between the predicted size of the target object and the size label of the target object is determined by using a preset bounding box sub-loss function, and a third sub-loss value is obtained;

[0027] The loss value between the predicted angle of the detection box and the angle label of the detection box is determined by using a preset angle classification sub-loss function, and a fourth sub-loss value is obtained.

[0028] The target loss value for this training is obtained by weighted summing of the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value.

[0029] In one possible implementation, the angle classification sub-loss function includes a Gaussian window function and a binary cross-entropy loss function; the step of determining the loss value between the predicted category of the target object and the category label of the target object using a preset target sub-loss function specifically involves:

[0030] Substituting the predicted angle into the Gaussian window function, the periodic angle value corresponding to the angle is obtained;

[0031] Substituting the periodic angle value and the angle label into the binary cross-entropy loss function, the loss value between the predicted angle of the detection box and the angle label of the detection box is obtained.

[0032] In one possible implementation, substituting the predicted angle into the Gaussian window function to obtain the periodic angle value corresponding to the angle specifically involves:

[0033] The predicted angle is processed by circular smoothing labels. When the predicted angle is within the window range of the Gaussian window function, the predicted angle is substituted into the Gaussian window function to obtain the periodic angle value corresponding to the angle.

[0034] When the predicted angle is not within the window range of the Gaussian window function, the angle is set to 0.

[0035] In one possible implementation, the box classification sub-loss function is the mean absolute value error loss function; the target sub-loss function is the triplet loss function; and the detection box sub-loss function is the cross-entropy loss function.

[0036] In a second aspect, the present invention provides a target detection device for fisheye images, the device comprising:

[0037] The first processing module is used to acquire fisheye images within a preset range of the road surface;

[0038] The second processing module is used to input the fisheye image into a pre-trained target detection model; the target detection model includes a convolutional neural network module, a rotated box target detection module, and a self-attention algorithm module;

[0039] The third processing module is used to identify target objects in the fisheye image using the convolutional neural network module, and to label the identified target objects using detection boxes;

[0040] The fourth processing module is used to rotate the detection box by a predicted angle using the rotating box target detection module, to obtain the rotating detection box with the least noise information as the target detection box, and output the first image; the first image is a fisheye image containing the target detection box;

[0041] The fifth processing module is used to enhance the feature region of the first image based on the background region of the first image using the self-attention algorithm module, and output the enhanced first image as the detection result; the background region is the region in the first image located outside the target detection box, and the feature region is the region located inside the target detection box.

[0042] In one possible implementation, the fourth processing module is specifically used to perform:

[0043] Obtain the position of the center point and the position of the long side of the detection frame;

[0044] Using the center point as the origin of the coordinate system and the straight line passing through the origin and parallel to the long side as the x-axis, the detection box is rotated clockwise around the center point by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box; the predicted angle ∈ [0°, 180°); the predicted angle is the angle between the x-axis of the detection box and the x-axis of the rotated detection box when the x-axis of the detection box and the long side of the rotated detection box begin to coincide.

[0045] In one possible implementation, the device further includes:

[0046] The sample data acquisition module is configured to acquire training samples, which include sample images and label information of target objects in the sample images. The label information includes the position label of the detection box of the target object, the angle label of the detection box, the category label of the target object, and the size label of the target object.

[0047] The input module is configured to input the sample image into an initial detection model, the initial detection model including an initial convolutional neural network module, an initial rotated box object detection module, and an initial self-attention algorithm module;

[0048] The data processing module is configured to perform target object detection in the sample image using the initial convolutional neural network module, and to label the target object with a detection box;

[0049] The detection box is rotated by a predicted angle using the initial rotated box target detection module to obtain the rotated detection box with the least noise information as the target detection box, and the first image is output.

[0050] The initial self-attention algorithm module is used to enhance the feature region of the first image based on the background region of the first image. The enhanced first image is output as the detection result, and the predicted position of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box are obtained.

[0051] Multiple sub-loss functions are used to determine the sub-loss values ​​between the predicted position, the predicted category, the predicted angle, the predicted size and the corresponding label information, and the target loss value for this training is determined based on the sub-loss values.

[0052] The model training module is configured to modify the parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module according to the target loss value, and continue training using the training samples until the loss value between the predicted feature information and the label information is less than a preset threshold, at which point training ends and an object detection model is obtained.

[0053] In one possible implementation, the target loss value determination module is specifically configured to perform:

[0054] The loss value between the predicted position of the detection box and the position label of the detection box is determined by using a preset detection box sub-loss function, and the first sub-loss value is obtained.

[0055] The loss value between the predicted category of the target object and the category label of the target object is determined by using a preset target sub-loss function, and a second sub-loss value is obtained;

[0056] The loss value between the predicted size of the target object and the size label of the target object is determined by using a preset bounding box sub-loss function, and a third sub-loss value is obtained;

[0057] The loss value between the predicted angle of the detection box and the angle label of the detection box is determined by using a preset angle classification sub-loss function, and a fourth sub-loss value is obtained.

[0058] The target loss value for this training is obtained by weighted summing of the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value.

[0059] In one possible implementation, the target loss value determination module is further configured to perform:

[0060] Substituting the predicted angle into the Gaussian window function, the periodic angle value corresponding to the angle is obtained;

[0061] Substituting the periodic angle value and the angle label into the binary cross-entropy loss function, the loss value between the predicted angle of the detection box and the angle label of the detection box is obtained.

[0062] In one possible implementation, the target loss value determination module is further configured to perform:

[0063] The predicted angle is processed by circular smoothing labels. When the predicted angle is within the window range of the Gaussian window function, the predicted angle is substituted into the Gaussian window function to obtain the periodic angle value corresponding to the angle.

[0064] When the predicted angle is not within the window range of the Gaussian window function, the angle is set to 0.

[0065] In one possible implementation, the target loss value determination module is further configured to perform:

[0066] The loss function for the bounding box is the mean absolute value error loss function; the loss function for the target is the triplet loss function; and the loss function for the detected bounding box is the cross-entropy loss function.

[0067] Thirdly, the present invention provides an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the target detection method for fisheye images as described above.

[0068] Fourthly, a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the target detection method for fisheye images described in any of the preceding claims.

[0069] The target detection method for fisheye images provided in this invention first acquires a fisheye image within a preset range of the road surface; second, the fisheye image is input into a pre-trained target detection model, which includes a convolutional neural network module, a rotated bounding box target detection module, and a self-attention algorithm module; third, the convolutional neural network module identifies target objects in the fisheye image and labels them using horizontal detection boxes; then, the rotated bounding box target detection module rotates the detection boxes by a predicted angle to obtain the rotated detection box with the least noise information as the target detection box, and the fisheye image containing the target detection box is output as the first image; finally, the self-attention algorithm module performs image enhancement processing on the feature regions located inside the target detection box based on the background region located outside the target detection box in the first image, to obtain the processing result. Compared to the traditional method of detecting target objects using horizontal bounding boxes, this scheme rotates the horizontal bounding boxes to obtain rotated bounding boxes. Compared to horizontal bounding boxes, rotated bounding boxes have a higher degree of fit with the target objects and can effectively reduce background information within the bounding boxes, thereby improving the target detection accuracy of fisheye images. Furthermore, the angle classification sub-loss function transforms the regression problem into a classification problem, solving the angle boundary problem and further improving the target object detection accuracy.

[0070] Compared with traditional methods that use convolutional neural networks to identify target objects, this invention introduces a self-attention mechanism, which can utilize the relationship between pixels in the feature region and pixels in the background region to help supplement the detailed information in the feature region, thereby enhancing the image of the feature region and making the image of the feature region in the obtained detection result clearer. Attached Figure Description

[0071] Figure 1 A flowchart illustrating the steps of a target detection method for fisheye images provided in an embodiment of the present invention;

[0072] Figure 2 A flowchart illustrating the steps of a training method for a target detection model provided in an embodiment of the present invention;

[0073] Figure 3 This is a structural block diagram of a target detection device for fisheye images provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0076] To address the issue of low target detection accuracy in fisheye images, this invention provides a target detection method and apparatus for fisheye images.

[0077] Figure 1 A flowchart illustrating the steps of a target detection method for fisheye images provided in an embodiment of the present invention.

[0078] like Figure 1 As shown, the target detection method for fisheye images includes:

[0079] Step 101: Obtain a fisheye image within a preset range of the road surface.

[0080] Specifically, while the vehicle is in motion, a fisheye camera is used to acquire fisheye images of the vehicle within a preset range on the road surface.

[0081] A fisheye camera refers to a camera equipped with a fisheye lens. A fisheye lens is a lens with an extremely short focal length and a field of view close to or equal to 180°, with a focal length of less than 16mm.

[0082] Fisheye cameras allow the lens to reach its maximum shooting angle, typically 220° or 230°, creating conditions for close-up shooting of large-scale scenes.

[0083] A fisheye image is an image taken by a fisheye camera. In a fisheye image, the objects in the center of the image are the same as those in a picture taken by a traditional camera, while objects in other parts of the image that should be horizontal or vertical are barrel-shaped distorted.

[0084] Fisheye images are playing an increasingly important role in intelligent vehicles due to their wide field of view, rich semantic information, and low cost. In this embodiment, fisheye images are used for detection and early warning during autonomous driving.

[0085] The preset range of the road surface can be within 5 meters or within 10 meters. This embodiment does not impose specific restrictions on the preset range of the road surface.

[0086] Step 102: Input the fisheye image into the pre-trained target detection model.

[0087] The object detection model includes a convolutional neural network module, a rotated bounding box object detection module, and a self-attention algorithm module.

[0088] Specifically, the collected fisheye images of vehicles within a preset range on the road surface are input into a pre-trained target detection model.

[0089] The core of the convolutional neural network module is the convolutional neural network, which is a type of feedforward neural network. It is mainly used in large-scale image processing scenarios, especially in the field of pattern classification. This network avoids complex preprocessing of images and can directly input the original image. That is, the input of the convolutional neural network module is a fisheye image of the road surface within a preset range obtained from the fisheye image.

[0090] In object detection, the commonly used horizontal bounding box is difficult to fit the target object, so a rotated bounding box is needed instead. The core of the rotated bounding box object detection module is the rotated bounding box object detection algorithm, which can convert horizontal bounding box object detection into rotated bounding box object detection.

[0091] The target object can be an obstacle, a pedestrian, another vehicle to be inspected, or any other object that may affect the normal operation of the vehicle.

[0092] The core of the self-attention algorithm module is the self-attention algorithm, also known as the self-attention mechanism, which enables the machine to notice the correlation between different parts of the input image.

[0093] Step 103: Use the convolutional neural network module to identify target objects in the fisheye image, and label the identified target objects using detection boxes.

[0094] When the target object is another vehicle, the vehicle in the fisheye image is identified by the convolutional neural network module, and the identified vehicle is labeled by the horizontal rectangular detection box. The labeled fisheye image containing the detection box is then used as the output of the convolutional neural network module.

[0095] Step 104: Use the rotating box target detection module to rotate the detection box by the predicted angle, obtain the rotating detection box with the least noise information as the target detection box, and output the first image.

[0096] The first image is a fisheye image containing the target detection bounding box. The prediction angle is the rotation angle at which the detection bounding box contains the least amount of noise information.

[0097] The labeled fisheye image containing the detection box is input into the rotating box target detection module. The rotating box target detection module rotates the detection box by a predicted angle with a point on or inside the detection box as the center to obtain the target detection box. The fisheye image containing the target detection box is output as the first image.

[0098] Step 105: Using the self-attention algorithm module, enhance the feature region of the first image based on the background region of the first image, and output the enhanced first image as the detection result.

[0099] The background region is the area outside the target detection box in the first image, and the feature region is the area inside the target detection box.

[0100] Specifically, the target detection box is a rectangular box that selects the target object. The self-attention algorithm module obtains the relationship between the pixels in the background region and the pixels in the feature region of the first image, and uses the image features of the background region to supplement or enhance the incomplete features of the feature region image, making the feature region image more complete and clear.

[0101] The target detection method for fisheye images provided in this invention first acquires a fisheye image within a preset range of the road surface; second, the fisheye image is input into a pre-trained target detection model, which includes a convolutional neural network module, a rotated bounding box target detection module, and a self-attention algorithm module; third, the convolutional neural network module identifies target objects in the fisheye image and labels them using horizontal detection boxes; then, the rotated bounding box target detection module rotates the detection boxes by a predicted angle to obtain the rotated detection box with the least noise information as the target detection box, and the fisheye image containing the target detection box is output as the first image; finally, the self-attention algorithm module performs image enhancement processing on the feature regions located inside the target detection box based on the background region located outside the target detection box in the first image, to obtain the processing result.

[0102] Compared to traditional methods of detecting target objects using horizontal detection boxes, this invention rotates the horizontal detection boxes to obtain rotated detection boxes. Compared to horizontal detection boxes, rotated detection boxes have a higher degree of fit with the target object and can effectively reduce background information within the detection box, thereby improving the target detection accuracy of fisheye images. Furthermore, the angle classification sub-loss function transforms the regression problem into a classification problem, solving the angle boundary problem and further improving the target object detection accuracy.

[0103] Compared with traditional methods that use convolutional neural networks to identify target objects, this invention introduces a self-attention mechanism, which can utilize the relationship between pixels in the feature region and pixels in the background region to help supplement the detailed information in the feature region, thereby enhancing the image of the feature region and making the image of the feature region in the obtained detection result clearer.

[0104] Furthermore, the detection box is rotated by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box, specifically including:

[0105] Obtain the position of the center point and the position of the long side of the detection box;

[0106] Using the center point as the origin and the straight line passing through the origin and parallel to the long side as the x-axis, rotate the detection box clockwise around the center point by a predicted angle to obtain the rotating detection box containing the least noise information as the target detection box; the predicted angle ∈ [0°, 180°); the predicted angle is the angle between the x-axis of the detection box and the x-axis of the rotating detection box when the x-axis of the detection box and the long side of the rotating detection box begin to coincide.

[0107] Furthermore, the detection box is rotated by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box, specifically including:

[0108] Obtain the position of the center point and the position of the long side of the detection box;

[0109] Using the center point as the origin and the straight line passing through the origin and parallel to the long side as the x-axis, the detection box is rotated clockwise around the center point by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box.

[0110] Wherein, the prediction angle ∈ [0°, 180°); the prediction angle is the angle between the x-axis of the detection box and the x-axis of the rotating detection box when the x-axis of the detection box and the long side of the rotating detection box begin to coincide.

[0111] Specifically, the detection box is a rectangle. A coordinate system is established with the center point of the rectangle as the origin and a straight line passing through the origin and parallel to the long side of the rectangle as the x-axis. The entire rectangle is rotated clockwise around the center point until the x-axis encounters the long side of the detection box before rotation. The angle between the x-axis before and after rotation is used as the prediction angle.

[0112] Figure 2 A flowchart illustrating the steps of a training method for a target detection model provided in an embodiment of the present invention.

[0113] like Figure 2 As shown, the object detection model is trained using the following method:

[0114] Step 201: Obtain training samples.

[0115] The training samples include sample images and label information of target objects in the sample images. The label information includes the location label of the detection box of the target object, the angle label of the detection box, the category label of the target object, and the size label of the target object.

[0116] The training samples consist of fisheye images of multiple objects in different modalities, each image in the training samples is labeled. Different modalities refer to the fact that the multiple fisheye images come from different imaging devices.

[0117] Specifically, each sample in the training samples includes the center point coordinates, length, and width of multiple labeled detection boxes, the rotation angle of the detection boxes, the category information of the target object, and the language image with the size information of the target object.

[0118] The target objects can be categorized as vehicles, trees, buildings, road barriers, and people.

[0119] Step 202: Input the sample image into the initial detection model.

[0120] The initial detection model includes an initial convolutional neural network module, an initial rotated bounding box object detection module, and an initial self-attention algorithm module. The output of the initial convolutional neural network module serves as the input to the initial rotated bounding box object detection module, and the initial rotated bounding box object detection module serves as the input to the initial self-attention algorithm module.

[0121] Step 203: Use the initial convolutional neural network module to detect target objects in the sample image and label the target objects with detection boxes.

[0122] Specifically, the initial convolutional neural network module identifies different target objects in the sample image, and different types of detection boxes are used to label the target objects.

[0123] Step 204: Rotate the detection box by the predicted angle using the initial rotated box target detection module to obtain the rotated detection box with the least noise information as the target detection box, and output the first image.

[0124] Specifically, sample images labeled with different types of detection boxes are input into the initial rotated box target detection module. The initial rotated box target detection module rotates the detection boxes to minimize the noise information contained in the rotated detection boxes, thereby obtaining the target detection boxes. The rotation angle of the target detection boxes relative to the detection boxes is obtained, and the sample image containing the target detection boxes is output as the first image.

[0125] Step 205: Use the initial self-attention algorithm module to enhance the feature region of the first image based on the background region of the first image, output the enhanced first image as the detection result, and obtain the predicted position of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box.

[0126] Specifically, the association between the pixels in the feature region and the pixels in the background region in the first image is obtained through the initial self-attention algorithm module, and the image is enhanced in the unclear or incomplete parts of the feature region by using the features in the background region that are associated with the feature region, so as to obtain the first image after the feature region has been enhanced.

[0127] Based on the first image after the feature regions have been enhanced, the predicted location of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box are obtained.

[0128] Step 206: Use multiple sub-loss functions to determine the sub-loss values ​​between the predicted position, predicted category, predicted angle, predicted size and the corresponding label information, and determine the target loss value for this training based on the sub-loss values.

[0129] The sub-loss values ​​between the predicted position, predicted category, predicted angle, predicted size and corresponding label information are determined by detecting the bounding box loss function, the target sub-loss function, the bounding box classification sub-loss function and the angle classification sub-loss function, respectively. Then, based on the determined sub-loss values, the target loss value for this training is determined.

[0130] Specifically, step 206 includes:

[0131] Step 2061: Use the preset detection box sub-loss function to determine the loss value between the predicted position of the detection box and the position label of the detection box, and obtain the first sub-loss value.

[0132] The detection box loss function is the cross-entropy loss function.

[0133] Specifically, cross-entropy represents the difference between the true probability distribution and the predicted probability distribution. The smaller the cross-entropy value, the better the model's prediction results.

[0134] Step 2062: Use the preset target sub-loss function to determine the loss value between the predicted category of the target object and the category label of the target object, and obtain the second sub-loss value.

[0135] In this embodiment, the target sub-loss function is the triplet loss function.

[0136] The triplet loss function is mainly used to distinguish between extremely similar but dissimilar samples. The triplet loss includes the loss between the image features of each image in a triplet image set in the training samples. The triplet image set includes: a reference image, a positive sample image, and a negative sample image. The reference image and the positive sample image are two images of the same object in different modalities, while the reference image and the negative sample image are two images of different objects in the same modality.

[0137] Step 2063: Use the preset box classification sub-loss function to determine the loss value between the predicted size of the target object and the size label of the target object, and obtain the third sub-loss value.

[0138] In this embodiment, the loss function for the box classification is the mean absolute value error loss function.

[0139] The mean absolute value error loss function, also known as the L1 loss function, is a commonly used regression loss function. It is the sum of the absolute values ​​of the difference between the target value and the predicted value, and is used to represent the average error magnitude of the predicted value.

[0140] For example, if the predicted target box is compared with each target box in the ground truth using L1 loss, and there are 20 target boxes in the ground truth, there will be 200 × 20 loss values.

[0141] Step 2064: Use the preset angle classification sub-loss function to determine the loss value between the predicted angle of the detection box and the angle label of the detection box, and obtain the fourth sub-loss value.

[0142] Among them, the angle classification sub-loss function includes the Gaussian window function and the binary cross-entropy loss function.

[0143] Substituting the predicted angle into the Gaussian window function yields the corresponding periodic angle value.

[0144] Specifically, the predicted angle is processed by circular smoothing labels. When the predicted angle is within the window range of the Gaussian window function, the predicted angle is substituted into the Gaussian window function to obtain the periodic angle value corresponding to the angle.

[0145] When the predicted angle is outside the window range of the Gaussian window function, the angle is set to 0.

[0146] Specifically, the predicted angle is processed using the following formula, transforming the angle regression problem into an angle classification problem.

[0147]

[0148] Here, g(x) is a window function, which is controlled by the window radius r.

[0149] The window function g(x) has four main properties:

[0150] Periodic g(x) = g(x + kT), k ∈ NT = 180 / ω (2)

[0151] Symmetry 0≤g(θ+ε)=g(θ-ε)≤1, |ε| <r (3)

[0152] The maximum value g(θ) = 1 (4)

[0153] Monotonicity: 0 ≤ g(θ ± ε) ≤ g(θ ± γ) ≤ 1, |γ| < |ε| <r(5)

[0154] Impulse functions, rectangular functions, trigonometric functions, and Gaussian functions can all be window functions that satisfy the above four properties.

[0155] In the angle classification loss function, the angle range is 0°-179°. Due to the periodicity of angles, the angle error at the boundaries is not as large as the actual numerical error. For example, the actual difference between 0° and 179° is not as large as the numerical difference. Therefore, a Gaussian function is chosen as the window function. Based on the periodicity of the angle, the true angle representation is given the same periodicity, making the true angle representation have a circular characteristic. This allows the loss function to achieve a large numerical difference at the calculation boundaries, but a small actual loss, reflecting the actual distance between each angle.

[0156] The Gaussian window function is as follows:

[0157]

[0158] Among them, A GT Let x be the true value of the angle, and let x range from -90° to 89°, μ = 0, σ = 6.

[0159] Substituting the periodic angle value and angle label into the binary cross-entropy loss function yields the loss value between the predicted angle of the detection box and the angle label of the detection box.

[0160] Specifically, for the predicted angle value, the angle loss is calculated using binary cross-entropy loss, where the binary cross-entropy loss function is:

[0161]

[0162] Among them, l angle This is the angle loss value. This is the predicted angle value. This is the true value of the angle.

[0163] Step 2065: Sum the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value with weights to obtain the target loss value for this training.

[0164] In this embodiment, the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value are assigned the same weight. That is, the target loss value for this training can be obtained by directly summing the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value.

[0165] Step 207: Modify the parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module according to the target loss value, and continue training using training samples until the loss value between the predicted feature information and the label information is less than the preset threshold. Training ends, and the object detection model is obtained.

[0166] Specifically, the parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module of the initial detection model are modified based on the target loss value obtained during training, until the loss value between the predicted feature information and the label information obtained by the model is less than a preset threshold. At this point, the model is considered to have achieved the desired effect, and the model training ends.

[0167] Furthermore, the initial detection model is constructed using the following method: the initial detection model includes a data loader module, a target feature extraction module, a combined target feature collection module, a prediction module, and a loss function module.

[0168] Specifically, the angle labels of the detection boxes are added to the label value data of the data loader of the convolutional neural network, and the data loader with the added angle labels is used as the data loading module of the initial detection model; the angle labels include the rotation angle of the detection boxes.

[0169] Specifically, the rotation angle of the detection box is between 0 and 180°. To achieve rotated box detection, each anchor box will rotate by a certain angle according to the long side definition method. The rotation angle information of the anchor box only exists in the angle label θ. Therefore, the dimension of the angle label θ needs to be added to the label value data of the data loader of the convolutional neural network.

[0170] In this model, the backbone network of a convolutional neural network is used as the target feature extraction module of the initial detection model.

[0171] Specifically, the target feature extraction module is used to extract features from the multi-scale features of the input image.

[0172] Among them, the combined target feature collection network layer of the convolutional neural network is used as the combined target feature collection module of the initial detection model.

[0173] Specifically, the combined target feature collection module is used to collect all feature information extracted from the same input image and display the collected combined features on the same image.

[0174] In the loss function part of the convolutional neural network, each angle within the rotation angle range is classified into a class by using a circular smooth label, and 180 angle classification channels from 0 to 179 are added to the loss function part to obtain the loss function module of the initial detection model.

[0175] Specifically, the angle label θ is added to the convolutional neural network to achieve the recognition of the rotating box of the detected target.

[0176] The self-attention algorithm module is used as the prediction module of the initial detection model.

[0177] Specifically, the prediction module of the convolutional neural network that processes feature data serially was changed to a self-attention algorithm module that processes feature data in parallel, thereby improving the processing speed of the prediction module for feature data and increasing the prediction accuracy of the new model obtained after the modification by 3 points.

[0178] For example, when the prediction module of a convolutional neural network predicts four feature information a1, a2, a3 and a4, the entire prediction process is executed sequentially. That is, the prediction result b1 is obtained by predicting a1 first, and then the prediction result b2 is obtained by predicting a2. The correlation between a1, a2, a3 and a4 is not considered.

[0179] The self-attention algorithm in this application processes the four feature information a1, a2, a3 and a4 in parallel, and predicts b1 based on a1 and the correlation between a1 and other features.

[0180] Figure 3 This is a structural block diagram of a target detection device for fisheye images provided in an embodiment of the present invention.

[0181] like Figure 3 As shown, the target detection device for fisheye images includes:

[0182] The first processing module 301 is used to acquire fisheye images within a preset range of the road surface;

[0183] The second processing module 302 is used to input the fisheye image into a pre-trained target detection model; the target detection model includes a convolutional neural network module, a rotated box target detection module, and a self-attention algorithm module;

[0184] The third processing module 303 is used to identify target objects in fisheye images using a convolutional neural network module and to label the identified target objects using detection boxes.

[0185] The fourth processing module 304 is used to rotate the detection box by a predicted angle using the rotating box target detection module, to obtain the rotating detection box with the least noise information as the target detection box, and output the first image; the first image is a fisheye image containing the target detection box.

[0186] The fifth processing module 305 is used to enhance the feature region of the first image based on the background region of the first image using the self-attention algorithm module, and output the enhanced first image as the detection result; the background region is the region in the first image located outside the target detection box, and the feature region is the region located inside the target detection box.

[0187] Furthermore, the fourth processing module 304 is specifically used to execute:

[0188] Obtain the position of the center point and the position of the long side of the detection box;

[0189] Using the center point as the origin and the straight line passing through the origin and parallel to the long side as the x-axis, rotate the detection box clockwise around the center point by a predicted angle to obtain the rotating detection box containing the least noise information as the target detection box; the predicted angle ∈ [0°, 180°); the predicted angle is the angle between the x-axis of the detection box and the x-axis of the rotating detection box when the x-axis of the detection box and the long side of the rotating detection box begin to coincide.

[0190] Furthermore, the device also includes:

[0191] The sample data acquisition module is configured to acquire training samples. The training samples include sample images and label information of target objects in the sample images. The label information includes the position label of the detection box of the target object, the angle label of the detection box, the category label of the target object, and the size label of the target object.

[0192] The input module is configured to input sample images into the initial detection model, which includes an initial convolutional neural network module, an initial rotated bounding box object detection module, and an initial self-attention algorithm module.

[0193] The data processing module is configured to perform target object detection in the sample image using the initial convolutional neural network module and label the target objects with detection boxes;

[0194] The target detection module uses an initial rotated bounding box to rotate the detection box by a predicted angle, obtaining the rotated detection box with the least noise information as the target detection box, and outputting the first image.

[0195] The initial self-attention algorithm module is used to enhance the feature region of the first image based on the background region of the first image. The enhanced first image is output as the detection result, and the predicted position of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box are obtained.

[0196] Multiple sub-loss functions are used to determine the sub-loss values ​​between the predicted position, predicted category, predicted angle, predicted size and the corresponding label information, and the target loss value for this training is determined based on the sub-loss values.

[0197] The model training module is configured to modify the parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module according to the target loss value, and continue training using training samples until the loss value between the predicted feature information and the label information is less than a preset threshold, at which point training ends and the object detection model is obtained.

[0198] Furthermore, the target loss value determination module is specifically configured to execute:

[0199] The loss value between the predicted position of the detection box and the position label of the detection box is determined by using a preset detection box sub-loss function, and the first sub-loss value is obtained.

[0200] The second sub-loss value is obtained by using a preset target sub-loss function to determine the loss value between the predicted category of the target object and the category label of the target object.

[0201] The loss value between the predicted size of the target object and the size label of the target object is determined by using a preset bounding box sub-loss function, thus obtaining the third sub-loss value;

[0202] The loss value between the predicted angle of the detection box and the angle label of the detection box is determined by using a preset angle classification sub-loss function, thus obtaining the fourth sub-loss value;

[0203] The target loss value for this training is obtained by weighted summing of the first, second, third, and fourth sub-loss values.

[0204] Furthermore, the target loss value determination module is further configured to perform:

[0205] Substituting the predicted angle into the Gaussian window function yields the corresponding periodic angle value.

[0206] Substituting the periodic angle value and angle label into the binary cross-entropy loss function yields the loss value between the predicted angle of the detection box and the angle label of the detection box.

[0207] Furthermore, the target loss value determination module is further configured to perform:

[0208] The predicted angle is processed by circular smoothing labels. When the predicted angle is within the window range of the Gaussian window function, the predicted angle is substituted into the Gaussian window function to obtain the periodic angle value corresponding to the angle.

[0209] When the predicted angle is outside the window range of the Gaussian window function, the angle is set to 0.

[0210] Furthermore, the target loss value determination module is further configured to perform:

[0211] The loss function for bounding box classification is the mean absolute error loss function; the loss function for target classification is the triplet loss function; and the loss function for detected bounding boxes is the cross-entropy loss function.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the target detection method for fisheye images in this invention.

[0214] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the target detection method for fisheye images in this invention.

[0215] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0216] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A target detection method for fisheye images, characterized in that, include: Acquire fisheye images within a preset range of the road surface; The fisheye image is input into a pre-trained target detection model; The target detection model includes a convolutional neural network module, a rotated bounding box target detection module, and a self-attention algorithm module; The convolutional neural network module is used to identify target objects in the fisheye image, and the identified target objects are labeled using detection boxes; The rotating box target detection module is used to rotate the detection box by a predicted angle to obtain the rotating detection box with the least noise information as the target detection box, and a first image is output; the first image is a fisheye image containing the target detection box; Using the self-attention algorithm module, the feature region of the first image is enhanced based on the background region of the first image, and the enhanced first image is output as the detection result. Specifically, the self-attention algorithm module obtains the relationship between the pixels in the background region and the pixels in the feature region of the first image, and uses the image features of the background region to supplement or enhance the incomplete features of the feature region image. The background region is the region in the first image located outside the target detection box, and the feature region is the region located inside the target detection box. The loss function of the target detection model is determined based on multiple sub-loss functions; the multiple sub-loss functions include a detection box sub-loss function, a target sub-loss function, a box classification sub-loss function, and an angle classification sub-loss function; the angle classification sub-loss function includes a Gaussian window function and a binary cross-entropy loss function, which are used to convert the predicted angle of the detection box into a periodic angle value before calculating the classification loss.

2. The method according to claim 1, characterized in that, The step of rotating the detection box by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box specifically includes: Obtain the position of the center point and the position of the long side of the detection frame; Using the center point as the origin of the coordinate system and the straight line passing through the origin and parallel to the long side as the x-axis, the detection box is rotated clockwise around the center point by a predicted angle to obtain the rotated detection box containing the least noise information as the target detection box; the predicted angle ∈ [0°, 180°); the predicted angle is the angle between the x-axis of the detection box and the x-axis of the rotated detection box when the x-axis of the detection box and the long side of the rotated detection box begin to coincide.

3. The method according to claim 1, characterized in that, The target detection model was trained using the following method: Acquire training samples, which include sample images and label information of target objects in the sample images. The label information includes the position label of the detection box of the target object, the angle label of the detection box, the category label of the target object, and the size label of the target object. The sample image is input into the initial detection model, which includes an initial convolutional neural network module, an initial rotated box target detection module, and an initial self-attention algorithm module. The initial convolutional neural network module is used to detect target objects in the sample image, and the target objects are labeled with detection boxes; The detection box is rotated by a predicted angle using the initial rotated box target detection module to obtain the rotated detection box with the least noise information as the target detection box, and the first image is output. The initial self-attention algorithm module is used to enhance the feature region of the first image based on the background region of the first image. The enhanced first image is output as the detection result, and the predicted position of the target detection box, the predicted category of the target object, the rotation angle of the target detection box, and the predicted size of the target detection box are obtained. Multiple sub-loss functions are used to determine the sub-loss values ​​between the predicted position, the predicted category, the predicted angle, the predicted size and the corresponding label information, and the target loss value for this training is determined based on the sub-loss values. The parameters of the initial convolutional neural network module, the initial rotated box object detection module, and the initial self-attention algorithm module are modified according to the target loss value, and training continues using the training samples until the loss value between the predicted feature information and the label information is less than a preset threshold. Training ends, and the object detection model is obtained.

4. The method according to claim 3, characterized in that, The method of using multiple sub-loss functions to determine the sub-loss values ​​between the predicted position, the predicted category, the predicted angle, the predicted size, and the corresponding label information specifically includes: The loss value between the predicted position of the detection box and the position label of the detection box is determined by using a preset detection box sub-loss function, and the first sub-loss value is obtained. The loss value between the predicted category of the target object and the category label of the target object is determined by using a preset target sub-loss function, and a second sub-loss value is obtained; The loss value between the predicted size of the target object and the size label of the target object is determined by using a preset bounding box sub-loss function, and a third sub-loss value is obtained; The loss value between the predicted angle of the detection box and the angle label of the detection box is determined by using a preset angle classification sub-loss function, and a fourth sub-loss value is obtained. The target loss value for this training is obtained by weighted summing of the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value.

5. The method according to claim 4, characterized in that, The angle classification sub-loss function includes a Gaussian window function and a binary cross-entropy loss function; the step of determining the loss value between the predicted angle of the detection box and the angle label of the detection box using the preset angle classification sub-loss function is specifically as follows: Substituting the predicted angle into the Gaussian window function, the periodic angle value corresponding to the angle is obtained; Substituting the periodic angle value and the angle label into the binary cross-entropy loss function, the loss value between the predicted angle of the detection box and the angle label of the detection box is obtained.

6. The method according to claim 5, characterized in that, The step of substituting the predicted angle into the Gaussian window function to obtain the periodic angle value corresponding to the angle is specifically as follows: The predicted angle is processed by circular smoothing labels. When the predicted angle is within the window range of the Gaussian window function, the predicted angle is substituted into the Gaussian window function to obtain the periodic angle value corresponding to the angle. When the predicted angle is not within the window range of the Gaussian window function, the angle is set to 0.

7. The method according to claim 4, characterized in that, The loss function for the bounding box classification is the mean absolute value error loss function; the loss function for the target is the triplet loss function; and the loss function for the detected bounding box is the cross-entropy loss function.

8. A target detection device for fisheye images, characterized in that, The device includes: The first processing module is used to acquire fisheye images within a preset range of the road surface; The second processing module is used to input the fisheye image into a pre-trained target detection model; the target detection model includes a convolutional neural network module, a rotated box target detection module, and a self-attention algorithm module; The third processing module is used to identify target objects in the fisheye image using the convolutional neural network module, and to label the identified target objects using detection boxes; The fourth processing module is used to rotate the detection box by a predicted angle using the rotating box target detection module, to obtain the rotating detection box with the least noise information as the target detection box, and output the first image; the first image is a fisheye image containing the target detection box; The fifth processing module is used to enhance the feature region of the first image based on the background region of the first image using the self-attention algorithm module, and output the enhanced first image as the detection result; the fifth processing module is specifically configured to perform: obtain the relationship between the pixels of the background region and the pixels of the feature region in the first image through the self-attention algorithm module, and use the image features of the background region to supplement or enhance the missing features of the feature region image. The background region is the region in the first image located outside the target detection box, and the feature region is the region located inside the target detection box; The loss function of the target detection model is determined based on multiple sub-loss functions; the multiple sub-loss functions include a detection box sub-loss function, a target sub-loss function, a box classification sub-loss function, and an angle classification sub-loss function; the angle classification sub-loss function includes a Gaussian window function and a binary cross-entropy loss function, which are used to convert the predicted angle of the detection box into a periodic angle value before calculating the classification loss.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the target detection method for fisheye images as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the target detection method for fisheye images as described in any one of claims 1-7.

Images