A data enhancement method, device and equipment for a target detection model

Abstract

The application provides a data enhancement method, device and equipment for a target detection model, comprising: determining a plurality of to-be-processed images by using the detection performance of the target detection model. In a preset image database, a target image matched with the to-be-processed image is determined, wherein the image database comprises a plurality of preset obstacle images pre-shot. A target mask of the target image is obtained, the target mask has an opacity channel, and smooth image fusion is realized. Image fusion is performed based on the target mask, the to-be-processed image and the target image to obtain a fused image. Weak samples requiring data enhancement are determined by using the target detection model, and directional data enhancement is performed. The to-be-processed image is fused with a pre-determined high-quality image, thereby solving the problems of lack of pertinence and low efficiency of a traditional data enhancement method, and the data enhancement effect can be improved more pertinently.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and specifically to a data augmentation method, apparatus, and device for a target detection model. Background Technology

[0002] In the field of environmental perception technology for autonomous driving systems, object detection is one of the key technologies for enabling vehicles to understand their surroundings and make behavioral decisions. Among these technologies, the detection of small obstacles (such as distant traffic cones, no-parking signs, and parking locks) has long been a major technical challenge for the industry. Because small objects occupy a small pixel area in an image and their extractable features are not obvious, their features are easily lost during downsampling, leading to low accuracy and a high false negative rate in object detection models. For autonomous driving systems, the problem of missing small objects directly impacts driving safety.

[0003] Furthermore, in autonomous driving datasets, the frequency and number of regular-sized targets (such as vehicles and pedestrians) are much higher than those of small targets. This imbalance in data distribution causes the target detection model to tend to learn the features of regular targets that are more numerous during training, while it fails to learn the features of small targets, which further deteriorates the detection performance of small targets.

[0004] Data augmentation can often be used to improve the generalization ability and robustness of a model. For example, image regions can be randomly cropped, geometrically transformed, or enhanced using advanced blending techniques, and then the augmented regions can be randomly pasted into arbitrary locations in the background image. However, these methods often rely on random strategies, lack specificity, and are difficult to effectively address the accuracy problem of object detection models for small objects. Summary of the Invention

[0005] In view of this, this application aims to provide a data augmentation method, apparatus, and device for object detection models, so as to more effectively augment the data used to train the object detection model and improve the accuracy of the object detection model.

[0006] In a first aspect, this application provides a data augmentation method for an object detection model, the method comprising: Multiple images to be processed were identified based on an object detection model; In a preset image database, a target image that matches the image to be processed is determined. The image database includes multiple preset obstacle images. Obtain the target mask of the target image; Image fusion is performed based on the target mask, the image to be processed, and the target image to obtain a fused image.

[0007] In one possible implementation, determining the target image that matches the image to be processed in a preset image database includes: Obtain first semantic information of the image to be processed, the first semantic information including at least one of the following: the category of the target in the image to be processed, the illumination information of the image to be processed, and the shooting angle of the image to be processed; The second semantic information of the preset obstacle image is obtained, and the second semantic information includes at least one of the following: the category of the preset obstacle in the preset obstacle image, the lighting information of the preset obstacle image, and the shooting angle of the preset obstacle image; Among the multiple pieces of second semantic information, determine the target semantic information that matches the first semantic information; The preset obstacle image corresponding to the target semantic information is determined as the target image.

[0008] In one possible implementation, when the image to be processed is an image captured by a fisheye camera, the step of image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Obtain the fusion position of the target image in the image to be processed; Based on the fusion location and the parameters of the fisheye camera, determine the orientation angle corresponding to the fusion location; Based on the orientation angle, fisheye distortion processing is performed on the target image to obtain a distorted image; The image is obtained by fusing the target mask, the image to be processed, and the distorted image.

[0009] In one possible implementation, the image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Calculate the first average brightness value of the image to be processed and the second average brightness value of the target image, respectively; The brightness of pixels in the target image is adjusted using the first average brightness value and the second average brightness value to obtain a brightness-adjusted image; The image is obtained by fusing the target mask, the image to be processed, and the brightness adjustment image.

[0010] In one possible implementation, the image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Determine the direction of the light source in the image to be processed; Affine transformation is performed on the target mask based on the light source direction to determine the shadow mask; The shadow mask is used to process the corresponding region of the image to be processed to obtain the processed image; The image is obtained by fusing the target mask, the processed image, and the target image.

[0011] In one possible implementation, the step of image fusion based on the target mask, the processed image, and the target image to obtain the fused image includes: Obtain the physical distance between the preset obstacle in the target image and the camera; The opacity coefficient is determined based on the negative correlation between the physical distance and the opacity coefficient. The image is obtained by fusing the opacity coefficient, the target mask, the processed image, and the target image.

[0012] In one possible implementation, determining multiple images to be processed based on the object detection model includes: The target detection model is used to detect targets in multiple autonomous driving images, and the prediction results and confidence scores of the predicted targets in each autonomous driving image are obtained, wherein the targets in the autonomous driving images have real labels; Based on the prediction results and the true labels, the loss value is determined; Based on the loss value and / or the confidence level, multiple autonomous driving images are filtered to determine multiple images to be processed.

[0013] In one possible implementation, obtaining the target mask of the target image includes: Instance segmentation is performed on the target image to obtain a binary mask of the target image; The binary mask is subjected to Gaussian blurring to obtain the target mask.

[0014] Secondly, this application provides a data augmentation apparatus for a target detection model, the apparatus comprising: The determination unit is used to determine multiple images to be processed based on the object detection model; A matching unit is used to determine a target image that matches the image to be processed in a preset image database, wherein the image database includes multiple preset obstacle images; The acquisition unit is used to acquire the target mask of the target image; The data enhancement unit is used to perform image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image.

[0015] Thirdly, this application provides an electronic device, the device comprising: a memory and a processor; The memory is used to store the relevant program code; The processor is used to call the program code to execute the data augmentation method for the target detection model as described in any of the implementations of the first aspect above.

[0016] Fourthly, this application provides a computer-readable storage medium for storing a computer program for executing the data augmentation method for a target detection model as described in any implementation of the first aspect above.

[0017] Fifthly, this application provides a computer program product comprising a computer program / instruction, which, when executed by a processor, implements the data augmentation method for a target detection model as described in any of the implementations of the first aspect.

[0018] In the above implementation of this application, in order to perform targeted data augmentation on the sample data of the trained object detection model, multiple images to be processed, i.e., images for targeted data augmentation, can first be determined using the detection performance of the object detection model. The object detection model can be used to detect obstacles in images of autonomous vehicles, such as traffic cones, no-parking signs, parking locks, and wheels. Target images matching the images to be processed are determined from a pre-set image database. This image database includes multiple pre-captured images of pre-defined obstacles, and these images have high image quality, allowing for data augmentation of the images to be processed. A target mask for the target image is obtained. This target mask has an opacity channel, enabling smooth image fusion. Image fusion is performed based on the target mask, the images to be processed, and the target image to obtain a fused image. The method provided in this application can utilize the detection performance of the object detection model to determine weak samples requiring data augmentation, i.e., the images to be processed, and perform targeted data augmentation. By using pre-determined high-quality images for matching and fusion with the images to be processed, the problems of lack of specificity and low efficiency in traditional data augmentation methods are solved, allowing for more targeted improvement of data augmentation effects. The object detection model was then trained further using the data-enhanced fused images to improve its accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments provided in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 A flowchart illustrating a data augmentation method for a target detection model provided in an embodiment of this application.

[0021] Figure 2a This is a schematic diagram of an image to be processed, provided as an embodiment of this application.

[0022] Figure 2b This is a schematic diagram of a target image provided in an embodiment of this application.

[0023] Figure 2c This is a schematic diagram of a fused image provided in an embodiment of this application.

[0024] Figure 3 A flowchart illustrating a method for training an object detection model, as provided in an embodiment of this application.

[0025] Figure 4 This is a schematic diagram of a data augmentation device for a target detection model provided in an embodiment of this application.

[0026] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely exemplary implementations of this application and not all implementation methods. Those skilled in the art can obtain other embodiments in conjunction with the embodiments of this application without creative effort, and these embodiments are also within the protection scope of this application.

[0028] In the field of environmental perception technology for autonomous driving systems, object detection is one of the key technologies for enabling vehicles to understand their surroundings and make behavioral decisions. Among these technologies, the detection of small obstacles (such as distant traffic cones, no-parking signs, and parking locks) has long been a major technical challenge for the industry. Because small objects occupy a small pixel area in an image and their extractable features are not obvious, their features are easily lost during downsampling, leading to low accuracy and a high false negative rate in object detection models. For autonomous driving systems, the problem of missing small objects directly impacts driving safety.

[0029] Furthermore, in autonomous driving datasets, the frequency and number of regular-sized targets (such as vehicles and pedestrians) are much higher than those of small targets. This imbalance in data distribution causes the target detection model to tend to learn the features of regular targets that are more numerous during training, while it fails to learn the features of small targets, which further deteriorates the detection performance of small targets.

[0030] Data augmentation can often be used to improve the generalization ability and robustness of a model. For example, image regions can be randomly cropped, geometrically transformed, or enhanced using advanced blending techniques, and then the augmented regions can be randomly pasted into arbitrary locations in the background image. However, these methods often rely on random strategies, lack specificity, and are difficult to effectively address the accuracy problem of object detection models for small objects.

[0031] Based on this, embodiments of this application provide a data augmentation method for an object detection model, enabling targeted augmentation of the data used to train the object detection model, improving the data augmentation effect, and enhancing the accuracy of the object detection model. Specifically, the detection performance of the object detection model is first used to identify multiple images to be processed, i.e., images specifically requiring data augmentation. The object detection model can be used to detect obstacles in images of autonomous vehicles, such as traffic cones, no-parking signs, parking locks, and wheels. In a preset image database, target images matching the images to be processed are identified. The image database includes multiple pre-captured images of preset obstacles, and these images have high image quality, allowing for data augmentation of the images to be processed. A target mask for the target image is obtained. This target mask has an opacity channel, enabling smooth image fusion. Image fusion is performed based on the target mask, the image to be processed, and the target image to obtain a fused image. Using the method provided in this application, the detection performance of the object detection model can be used to identify weak samples requiring data augmentation, i.e., the images to be processed, for targeted data augmentation. By matching and fusing pre-determined high-quality images with the image to be processed, the problems of traditional data augmentation methods—lack of specificity and low efficiency—are solved, allowing for more targeted improvement of data augmentation results. The fused image with augmented data is then used to further train the object detection model, thereby improving its accuracy.

[0032] To facilitate understanding of the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings.

[0033] See Figure 1 The diagram shown is a flowchart of a data augmentation method for a target detection model provided in an embodiment of this application.

[0034] Alternatively, this method can be executed by a data processing device. This data processing device can be a terminal device or a server.

[0035] The method may include the following steps: S101: Identify multiple images to be processed based on the object detection model.

[0036] The object detection model can represent a pre-trained model capable of detecting objects in autonomous driving scenarios. For example, the object detection model could be a convolutional neural network, a YOLO model, etc. To specifically augment the samples used to train the object detection model, the model's detection performance on the samples can be combined to identify images with weak detection performance that require data augmentation, i.e., the images to be processed.

[0037] In practice, the images to be processed can be obtained as follows: Object detection models are used to detect objects in multiple autonomous driving images, and the prediction results and confidence scores of the objects in each autonomous driving image are obtained. The autonomous driving images can represent images of the autonomous driving scene captured by the vehicle during autonomous driving using front-view cameras, rear-view cameras, or surround-view cameras. Each autonomous driving image includes one or more objects, representing obstacles around the vehicle in the autonomous driving scene. For example, objects can include other vehicles, pedestrians, traffic cones, no-parking signs, parking locks, etc. Each object in the autonomous driving image has a pre-labeled ground truth label, which can include: the object's ground truth category, bounding box, etc. The bounding box can be represented by four vertex coordinates, or by length, width, and center point coordinates, used to determine the object's position and size.

[0038] The autonomous driving image is input into the object detection model. The model determines the prediction result and confidence score of one or more predicted objects in the autonomous driving image. The prediction result includes: the predicted category of the predicted object and the predicted bounding box (used to determine the position and size of the predicted object). The confidence score indicates the likelihood that the predicted object belongs to the predicted category; the higher the confidence score, the higher the accuracy of the predicted category. Then, based on the prediction result and the corresponding ground truth label, a loss value is determined. For example, a category loss value can be determined based on the ground truth category of the object and the corresponding predicted category of the predicted object, and a position loss value can be determined based on the object's bounding box and the corresponding predicted bounding box of the predicted object. The category loss value and the position loss value are then weighted and fused to obtain the loss value for each predicted object. The larger the loss value, the less accurate the prediction result, meaning the worse the object detection model's performance in detecting autonomous driving images.

[0039] Multiple autonomous driving images are filtered based on their corresponding loss value and / or confidence score to determine several images to be processed. Since these images require data augmentation, autonomous driving images with poor object detection model performance can be selected as the processing images. Lower confidence scores result in lower accuracy in predicting the target's class, leading to less accurate predictions from the object detection model. Similarly, higher loss scores result in less accurate predictions. Therefore, autonomous driving images with larger loss scores or lower confidence scores can be selected as the processing images.

[0040] In practical implementation, since each autonomous driving image may include one or more predicted targets, the loss value and confidence level corresponding to each predicted target can be obtained. For all predicted targets in each autonomous driving image, the total loss value corresponding to all predicted targets can be calculated. Then, multiple autonomous driving images are sorted in descending order of total loss value, and the autonomous driving image with the highest pre-defined proportion is selected as the image to be processed.

[0041] Alternatively, for each autonomous driving image, the number of predicted targets with a confidence level less than a confidence threshold can be counted. Then, the multiple autonomous driving images are sorted in descending order of the number of predicted targets, and the second-highest ranked autonomous driving image is selected as the image to be processed.

[0042] Alternatively, the two methods can be combined, using the top-ranked autonomous driving images by total loss value and the top-ranked images by quantity as both the images to be processed. This method selects images with poor object detection model performance as those requiring data augmentation, allowing the augmented images to be used to further train the object detection model and specifically improve its detection performance.

[0043] It should be noted that the specific values ​​of the first preset ratio, the second preset ratio, and the confidence threshold can be set according to actual needs and scenarios, and this application embodiment does not limit them. For example, the first preset ratio and the second preset ratio can be set to 20%, and the confidence threshold can be set to 0.3.

[0044] By utilizing the performance feedback of the object detection model to determine the image to be processed, subsequent image fusion is no longer random, but directly supplements the data for small target obstacles that are difficult for the object detection model to detect, thus solving the problem of low training efficiency caused by the randomness of data augmentation.

[0045] S102: In a preset image database, determine the target image that matches the image to be processed. The image database includes multiple preset obstacle images.

[0046] Target detection models often exhibit poor detection performance for small obstacles in autonomous driving images, such as distant traffic cones, unlocked parking locks, bollards, and no-parking signs. Optionally, in this embodiment, the image size of the small obstacle can be set to less than 32x32 pixels. Due to factors such as distance, lighting, and weather conditions, small obstacles are often poorly imaged in autonomous driving images, easily leading to missed or false detections by the target detection model. To improve the detection performance of the target detection model for these small obstacles, images of various preset obstacles can be pre-captured under different lighting conditions, weather conditions, and shooting angles to form an image database. These preset obstacle images should have high clarity and resolution. The preset obstacles refer to the aforementioned small obstacles.

[0047] To perform targeted data augmentation on the image to be processed, a target image matching the image to be processed can be identified from multiple preset obstacle images. This target image can then be fused with the image to be processed to achieve better data augmentation results. In one possible implementation, first semantic information of the image to be processed can be obtained. This first semantic information includes at least one of the following: the category of the target in the image to be processed, the illumination information of the image to be processed, and the shooting angle of the image to be processed. The category of the target can be obtained through the target's ground truth label, which includes the target's true category and a bounding box. The bounding box is used to determine the target's position and size. The illumination information of the image to be processed can be represented by statistically analyzing the brightness information of the image to be processed. The shooting angle of the image to be processed can be directly obtained from the parameters of the camera that captured the image to be processed.

[0048] Similarly, second semantic information of the preset obstacle image can be obtained. This second semantic information includes at least one of the following: the category of the preset obstacle in the preset obstacle image, the lighting information of the preset obstacle image, and the shooting angle of the preset obstacle image. Since the preset obstacle image is obtained by photographing known preset obstacles, the category of the preset obstacle is known. The lighting information of the preset obstacle image can be represented by statistically analyzing the brightness information of the preset obstacle image. The shooting angle of the preset obstacle image can be directly obtained from the parameters of the camera that captured the preset obstacle image.

[0049] Since the image database includes multiple preset obstacle images, multiple sets of second semantic information can be obtained. Then, the first semantic information is matched against each of the multiple sets of second semantic information to determine the second semantic information that matches the first semantic information as the target semantic information. Specifically, matching can be performed based on the category, lighting information, and shooting angle in the first semantic information, and the category, lighting information, and shooting angle in the second semantic information. If no completely matching second semantic information is found, matching can be performed based on the category and lighting information in the first semantic information, and the category and lighting information in the second semantic information. The preset obstacle image corresponding to the target semantic information is then determined as the target image that matches the image to be processed.

[0050] It should be noted that the types of the first semantic information and the second semantic information provided in the above embodiments are not limited to the above forms, and may also include weather conditions, road conditions, etc., which are not limited in this application embodiment.

[0051] Optionally, the first semantic information may further include the size of the target in the image to be processed, which can be obtained through the bounding box corresponding to the target. The second semantic information may further include the size of a preset obstacle. For example, the size of the preset obstacle can be detected and determined using image segmentation, target detection, or other methods. Based on this, after determining the preset obstacle image corresponding to the target semantic information, the preset obstacle in the preset obstacle image can be scaled and adjusted based on the size of the target in the image to be processed, so as to maximize the size similarity between the preset obstacle and the target. By selecting a target image that is semantically similar to the image to be processed, the final generated fused image can be closer to the real physical scene, avoiding the physical distortion problem caused by data augmentation.

[0052] S103: Obtain the target mask of the target image.

[0053] The target mask is used to distinguish pixel regions in the target image that belong to preset obstacles from pixel regions that belong to the background, in order to achieve more refined image fusion. In this embodiment, to make the image fusion effect more natural, the target mask represents a mask with a smooth alpha channel, making the edge transitions smoother and more natural.

[0054] In one possible implementation, the target mask of the target image can be obtained through the following steps: First, the target image can be segmented to obtain a binary mask. For example, a deep learning-based instance segmentation model can be used to segment the target image pixel by pixel, resulting in a binary mask where the pixel value of the foreground region (the region containing the preset obstacle) is 1 and the pixel value of the background region (the region not containing the preset obstacle) is 0. Then, the binary mask... Perform Gaussian blurring to obtain the target mask. The Gaussian blurring process can be expressed by the formula: Among them, standard deviation Based on the preset obstacle pixel width w_ 预设 Dynamic adjustment. For example, it can be represented as =w_ 预设 / 10. At this point, the pixel values ​​in the target mask can be understood as opacity Alpha. The larger the opacity Alpha, the higher the opacity.

[0055] By applying Gaussian blur to the binary mask, a region with pixel values ​​transitioning continuously from 1 to 0 can be generated at the edge of the binary mask, avoiding jagged edges in the target mask. When subsequently using the target mask to blend preset obstacles into the image to be processed, a smooth edge transition is achieved, eliminating any disjointedness that may occur during fusion and improving the texture-like appearance issues that arise during image fusion.

[0056] S104: Based on the target mask, fuse the image to be processed and the target image to obtain a fused image.

[0057] Once the target image and target mask that match the image to be processed are obtained, the image fusion operation can be performed to blend the preset obstacles in the target image into the background of the image to be processed.

[0058] In practice, the target image and the image to be processed can be fused together by considering the fusion position of the target image within the image to be processed. Optionally, the fusion position of the target image can be determined based on the position of the target in the image to be processed. Alternatively, the fusion position can be determined by appropriately moving the target based on its position, ensuring that the moved position is in a blank area of ​​the image to be processed. Since the target image includes pre-set obstacles, this fusion position can serve as the grounding point when fusing the pre-set obstacles into the image to be processed. Based on this, after determining the fusion position, given the size of the target image, the area occupied by the target image when fused into the image to be processed can be determined, which is the fusion region. Therefore, fusing the image to be processed and the target image is essentially fusing the fusion region of the image to be processed with the target image.

[0059] In practical implementation, it can be done as follows: This represents the image to be processed. Represents the target image. Represent the target mask, and use the target mask Blending region of the image to be processed and target image Processing is performed to obtain the target area. As the region in the fused image corresponding to the fused region, it can be represented as: Image to be processed The regions other than the fusion region remain unchanged, and the other regions of the fused image are composed of the target region and the fusion region. Combine to obtain a fused image .

[0060] Optionally, the above fusion formula can be expressed as utilizing The RGB channel values ​​of each pixel in the fusion region are compared with those of the target image. The RGB channel values ​​of the corresponding pixels are weighted and fused to obtain the target area. The RGB channel values ​​of the corresponding pixel in the image to be processed. The pixel values ​​in areas other than the fused region remain unchanged.

[0061] In practical applications, when fusing the image to be processed and the target image, it is also necessary to adapt to the shooting effects of different types of cameras. For example, images captured by fisheye cameras have a certain degree of distortion. In one possible implementation, when the image to be processed is an image captured by a fisheye camera, it has certain distortion characteristics. In order to make the preset obstacles in the target image blend more naturally into the image to be processed, the distortion characteristics of the fisheye lens can be simulated to distort the target image, and then the distorted image can be fused into the image to be processed.

[0062] In practice, the following steps A1-A4 may be included: A1: Obtain the fusion position of the target image in the image to be processed.

[0063] The fusion position of the target image can be determined based on the location of the target in the image to be processed. Alternatively, the target can be moved appropriately to serve as the fusion position, ensuring that the moved position is located in a blank area of ​​the image to be processed. This fusion position can serve as the grounding point of a preset obstacle in the target image. The grounding point of the preset obstacle can be understood as: determining the bounding box of the preset obstacle, with the midpoint of the bottom edge of the bounding box being the grounding point.

[0064] A2: Determine the orientation angle corresponding to the fusion location based on the fusion location and the parameters of the fisheye camera.

[0065] This orientation angle can be understood as the orientation angle of a preset obstacle in the real world when captured by a fisheye camera at the fusion location. The orientation in the real world can be calculated from the position of the preset obstacle in the image to be processed.

[0066] In practice, the orientation angle can be determined as follows: the fusion position of the preset obstacle in the target image in the image to be processed is represented as (x, y), and the orientation angle at the fusion position is represented as... First, obtain the calibration parameters of the fisheye camera, including intrinsic parameters, extrinsic parameters, focal length, principal point, and distortion coefficients. The intrinsic parameters are represented as an intrinsic parameter matrix. , Indicates focal length (in pixels), (The coordinates of the principal point are ) and the distortion coefficient is . (This represents the relationship between the distortion radius and the incident angle). The extrinsic parameters of a fisheye camera are determined by the rotation matrix. Translation vector express.

[0067] The specific calculation process includes the following steps B1-B4: B1: Pixel Distortion and Normalization: Using Distortion Coefficients The coordinates (x, y) of the fusion location are then distorted. First, the distortion radius is calculated. Then, based on the fisheye projection model (such as equidistant projection) or polynomial Solve for the angle of incidence. This leads to the normalized coordinates (u, v) after distortion correction: .

[0068] B2: Back-projection ray: In the camera coordinate system, the direction vector corresponding to this normalized coordinate is... Transform to the vehicle coordinate system to obtain the ray direction from the camera's optical center to the preset obstacle. .

[0069] B3: Ground Intersection Solution: Assume the pre-defined obstacle is placed on a flat surface. The position of the camera's optical center in the vehicle coordinate system is... The parametric equation for a ray is expressed as: t is a proportional parameter. Let it be... The component is zero, that is Solving for: . and The value of t has already been obtained from the above formula, so it can be calculated. Substituting this value into the formula yields the grounding point of the preset obstacle. coordinates in the vehicle coordinate system Then it can be calculated that .

[0070] B4: Derived Orientation Angle: To ensure a natural visual effect after the preset obstacles are blended, their front should face the camera's projection center. The front orientation vector is defined as: the projection point on the ground from the ground point towards the camera's optical center. That is, the front direction vector Orientation Angle For the frontal direction vector and the vehicle The angle between the axis (directly in front) and the axis is calculated using the arctangent function: .

[0071] In one possible implementation, to avoid overfitting and increase data diversity, it is also possible to... Introducing random perturbation angles , as the final orientation angle. Among them, It can be determined from a preset perturbation angle range, which is expressed as: At this point, the final orientation angle can be expressed as... It should be noted that the settings can be configured according to the type of preset obstacle. The specific value, for example, when the preset obstacle is a ground lock, is set to... When the preset obstacle is a movable cone, set... .

[0072] A3: Perform fisheye distortion processing on the target image based on the orientation angle to obtain a distorted image.

[0073] After determining the orientation angle of the preset obstacle, we can also know the direction in which the fisheye camera captured the preset obstacle. Based on the orientation angle, we perform fisheye distortion processing on the target image to obtain a distorted image, which is essentially the distortion effect produced when the fisheye camera captured the preset obstacle. This ensures that the preset obstacle and the image to be processed have the same distortion effect, making the image resulting from their fusion more realistic.

[0074] In practice, the target image can be processed to remove fisheye distortion using the following steps C1-C3: C1: World Pose Modeling: Treat the preset obstacle as a rectangle perpendicular to a vertical plane on the ground, with a true height of... (meters), width is (meters) (provided by the image database). The default obstacle's contact point coordinates in the vehicle coordinate system are... Define the horizontal projection of the frontal normal vector of the preset obstacle and the vehicle. The included angle of the axis is The four corner points of the pre-defined obstacle in the vertical plane are represented as follows: , , , The coordinates of each corner point can be determined in the following way: C2: Fisheye camera projection: Projecting the four corner points onto the image coordinate system respectively. The steps include C21-C25: C21: Transform any corner point P from the vehicle coordinate system to the camera coordinate system. The coordinates in the camera coordinate system are then expressed as follows: ,in, ; C22: Normalized pinhole projection: , Calculate radial distance Angle of incidence .

[0075] C23: Application of fisheye distortion correction: based on distortion coefficient Calculate the distortion radius .

[0076] C24: Normalized coordinates after distortion: , .

[0077] C25: Internal parameter mapping: , This yields the pixel coordinates of the projection point of corner point P in the image coordinate system. .

[0078] This yields the projection points in four image coordinate systems. They form a quadrilateral on the image that reflects fisheye distortion and perspective contraction.

[0079] C3: Solving the perspective transformation matrix: transforming the target image The pixel width is represented as The height is The homogeneous coordinates of its four corner points are respectively , , , Using four sets of corresponding point pairs A 3x3 perspective transformation matrix can be solved using Direct Linear Transformation (DLT). This ensures that for each pair of corresponding points, there is (in Represents homogeneous coordinates. (This is the normalized scaling factor).

[0080] Perspective transformation formula: Use the warpPerspective function on the target image. Apply perspective transformation matrix Generate distorted images , ,in, Represents the target image The coordinates of any pixel in the ( ) represents a distorted image The coordinates of any pixel in the image. This represents the normalization factor for perspective transformation.

[0081] By distorting the target image, the pre-set obstacles and the image to be processed have the same visual effect, reducing the "texture-like" appearance of the fused image.

[0082] A4: Based on the target mask, the image to be processed and the distorted image are fused to obtain the fused image.

[0083] After distorting the target image, the image to be processed and the distorted image can be fused using a target mask to obtain a fused image. The specific fusion process can be found in the description of step S104 in the above embodiments, and will not be elaborated upon here.

[0084] As can be seen from the above embodiments, although a target image matching the image to be processed is selected in advance, there are still differences between the physical environment of the target image and the physical environment of the image to be processed. In order to further optimize the realism of image fusion, the embodiments of this application can also adjust the lighting conditions of the target image to make it more consistent with the lighting conditions of the image to be processed, thereby improving the image fusion effect.

[0085] In one possible implementation, after obtaining the target image, a first average brightness value of the image to be processed and a second average brightness value of the target image are calculated separately. Specifically, the image to be processed can be converted from RGB space to LAB space to obtain the brightness value of the L channel corresponding to each pixel. Then, the average brightness value of each pixel in the image to be processed is calculated, which is the first average brightness value. Similarly, the second average brightness value of the target image can be obtained using the same calculation method.

[0086] Then, using the first and second average brightness values, the brightness of each pixel in the target image is adjusted to obtain a brightness-adjusted image. This allows the image to be fused with the brightness-adjusted image based on the target mask, resulting in a fused image.

[0087] Specifically, the brightness of the target image can be adjusted in the following ways: This represents the average brightness of the first value. This represents the average brightness of the second value. This represents the brightness value of any pixel in the target image. This indicates the adjusted brightness value. It can be determined using the following formula: In this formula, k is the fine-tuning coefficient (usually between 0.9 and 1.1), and clamp is the truncation function that limits the brightness value to the range of 0 to 255. After obtaining the adjusted brightness value for each pixel, it is then converted to RGB space to obtain the brightness-adjusted image.

[0088] Alternatively, the brightness of the target image and the image to be processed can be matched by converting the RGB space to the HSV space and adjusting the brightness value. The above embodiments are merely illustrative examples, and other methods to achieve the same technical effect are also within the scope of protection of this application.

[0089] In practical application scenarios, assuming that a preset obstacle is located in the image to be processed, a shadow may be generated when the preset obstacle is illuminated by a light source. In order to make the target image and the image to be processed more closely resemble the real scene after fusion, the shadow of the preset obstacle in the image to be processed can also be generated.

[0090] One possible implementation mainly includes the following steps D1-D3: D1: Determine the direction of the light source for the image to be processed.

[0091] Specifically, the image to be processed can be determined. The brightness value of each pixel is calculated, and then multiple target pixels with high brightness values ​​are selected. The average pixel coordinates of these multiple target pixels are calculated, and this average pixel coordinates can be used as a reference point for the light source. The number of target pixels is not limited in this embodiment and can be determined based on the application scenario. For example, it can be 1%-5% of the total number of pixels.

[0092] D2: Perform an affine transformation on the target mask based on the light source direction to determine the shadow mask.

[0093] Here, the target mask represents the mask of the target image. Therefore, the target mask is subjected to affine processing, which means that the preset obstacles in the target image are subjected to affine processing.

[0094] Specifically, this can be achieved in the following ways: D21: Calculation of shadow projection direction: First, the fusion position of the target image in the image to be processed is obtained, which serves as the grounding point of the preset obstacle in the image coordinate system, i.e., the fusion position, denoted as... For example, the fusion location can be determined based on the position of the target in the image to be processed. Then, the shadow projection direction vector is calculated. This vector is defined as the opposite direction from the light source to the preset obstacle: .in, Represents the Euclidean distance between vectors. and These are the projection components of the direction vector onto the horizontal and vertical axes of the image, respectively.

[0095] D22: Affine Transformation Matrix Construction and Shadow Mask Generation: To simulate the perspective distortion of shadows on the ground, an affine transformation function is introduced. It is defined as an image Apply Transformation matrix Pixel coordinates are changed through linear mapping. This embodiment of the application is based on the shadow projection direction. Constructing the transformation matrix : The parameters are defined as follows: shear and scaling intensity coefficients. The estimated pixel height of the preset obstacle in the target image Proportional, let ,in The preset scaling factor ( Directional components It can be determined by the shadow projection direction vector. This allows for the control of the horizontal slant direction and the vertical scaling degree, respectively.

[0096] Translation parameters : Used to align the root of the shadow with the grounding point of a preset obstacle. The calculation process includes: initial setting Construct the initial transformation matrix Binary mask for the target image Perform an affine transformation to obtain the initial affine mask. .Pick The coordinates of the midpoint of the white pixels in the bottom row Then the translation parameter , It can be represented as: This determines the transformation matrix A.

[0097] Using transformation matrix A to create a binary mask Apply an affine transformation to obtain the shadow mask. : .

[0098] D3: Use shadow masks to process the corresponding areas of the image to be processed, and obtain the processed image.

[0099] Since the shadow mask is generated from the target image, when fusing the target image with the image to be processed, only the region in the image to be processed that corresponds to the target image will be processed, i.e., the fusion region in the above embodiment. Therefore, the shadow mask can be used to process the fusion region of the image to be processed, while other regions of the image to be processed remain unchanged, resulting in the processed image.

[0100] Optionally, to smooth the edges of the shadows, the shadow mask can be Gaussian blurred to obtain a Gaussian shadow mask. Then, the Gaussian shadow mask is used to process the fusion region of the image to be processed, resulting in the processed image.

[0101] Specifically, with This indicates the fusion region of the image to be processed. This indicates the merged region after processing. If the Gaussian shadow mask is used, then the processed blended region... Represented as: .in, This represents the shadow darkening intensity coefficient, with a value ranging from 0.3 to 0.7, and is used to control the degree of shadow darkening. The higher the value, the deeper the shadow. The value can be adjusted based on the scene brightness of the image to be processed; for example, a higher value is used under strong light, and a lower value is used on cloudy days. The pixel values ​​of other regions in the image to be processed, except for the fusion region, remain unchanged. The processed image is obtained by combining these other regions with the processed fusion region.

[0102] Then, the processed image and the target image are fused using a target mask to obtain a fused image. The specific implementation process can be found in the embodiment corresponding to step S104 above, and will not be elaborated further here.

[0103] Optionally, in order to further optimize the visual effect of image fusion in the above embodiments, the atmospheric perspective effect in real scenes can be simulated to make targets at different distances present different clarity differences, making the fused image more realistic.

[0104] In one possible implementation, after obtaining the processed image including shadows, the physical distance between a preset obstacle in the target image and the camera can be obtained. For example, if the target image is captured using a depth camera, the physical distance can be determined based on the depth information of the target image. If it is captured using a monocular camera, it can be obtained through monocular geometric reasoning, mainly including the following two methods: (1) Based on pixel height and prior physical size: First, obtain the actual height H of the preset obstacle in the target image (e.g., approximately 0.7 meters for a traffic cone and approximately 0.3 meters for a parking lock). Then, extract the bounding box of the preset obstacle from the target image to obtain its pixel height. By combining the intrinsic focal length F of the camera that captured the target image, the physical distance can be calculated using the relationship between similar triangles. : .

[0105] (2) Based on the assumption of ground plane and camera installation height: Given the camera's height above the ground and pitch angle In this scenario, if the ordinate of the grounding point of the preset obstacle is y, and the ordinate of the center of the target image is... The camera's focal length in the vertical direction is The physical distance can then be calculated using inverse perspective mapping. : This method is applicable to autonomous driving scenarios with flat ground and can more accurately reflect the spatial location of the target grounding point.

[0106] After obtaining the physical distance between the preset obstacle in the target image and the camera, the opacity coefficient is determined based on the negative correlation between the physical distance and the opacity coefficient. The opacity coefficient is used to adjust the transparency when the target image is fused into the image to be processed. Since the greater the physical distance, the less clearly the preset obstacle appears in the image to be processed, and thus the smaller the opacity coefficient, the negative correlation between the physical distance and the opacity coefficient can be predetermined. For example, an atmospheric transmission model can be introduced to define the opacity coefficient. , can be represented as: ,in, This represents the opacity coefficient. Atmospheric attenuation coefficient, Indicates physical distance. It decreases as d increases.

[0107] Finally, the processed image and the target image can be fused based on the opacity coefficient and the target mask to obtain a fused image. Optionally, the fusion position of the target image can be determined based on the position of the target in the image to be processed. This fusion position can serve as a grounding point for a preset obstacle to be fused into the image to be processed, thus determining the corresponding fusion region of the target image in the image to be processed.

[0108] Specifically, with This represents the image to be processed. Represents the target image. Indicates the target mask. This represents the processed image. This indicates the fusion region of the image to be processed. This represents the region in the processed image that corresponds to the fused region. Utilizing a target mask. and opacity coefficient The region corresponding to the fusion region in the processed image and target image The fusion process is performed to obtain the region in the fused image that corresponds to the fused region, which is represented as the target region. Target area This can be expressed by the following formula: The processed image contains, except for Other regions remain unchanged, and are considered as other regions in the fused image, i.e., regions other than those in the processed image. Other areas and target areas The images are combined to obtain a fused image.

[0109] It should be noted that the above embodiments describe various methods for processing the target image before image fusion, including fisheye distortion correction (when the image to be processed is taken by a fisheye camera), brightness adjustment, shadow processing, and opacity blending. It is understood that one or more of these methods can be combined to process the target image, thereby improving the similarity between the target image and the image to be processed, making the fused image more realistic and improving the data augmentation effect. For example, after executing steps A1-A3, continuing with one or more of steps A41-A44 yields the fused image: A1: Multiple images to be processed are determined based on the object detection model; A2: In the preset image database, determine the target image that matches the image to be processed. The image database includes multiple preset obstacle images; A3: Obtain the target mask of the target image; A41: When the image to be processed is an image taken by a fisheye camera, fisheye distortion processing is performed on the target image to obtain a distorted image; image fusion is performed based on the target mask, the image to be processed, and the distorted image to obtain a fused image; A42: Based on the brightness values ​​of the image to be processed and the brightness values ​​of the target image, the brightness of the pixels in the target image is adjusted to obtain a brightness-adjusted image; the target mask, the image to be processed, and the brightness-adjusted image are fused to obtain a fused image; A43: Perform an affine transformation on the target mask to obtain a shadow mask; use the shadow mask to process the corresponding region of the image to be processed to obtain a processed image; perform image fusion based on the target mask, the processed image, and the target image to obtain a fused image; A44: Determine the opacity coefficient based on the physical distance between the preset obstacle in the target image and the camera; perform image fusion based on the opacity coefficient, target mask, image to be processed, and target image to obtain a fused image.

[0110] This application provides a data augmentation application scenario. See details below. Figure 2a , 2b As shown in 2c, Figure 2a This is a schematic diagram of an image to be processed. Figure 2b This is a schematic diagram of a target image. Figure 2c This is a schematic diagram of image fusion. The image to be processed is taken while a vehicle is driving in a garage, and the target image is taken with a tripod. The image to be processed and the target image are fused to obtain... Figure 2c The fused image shown.

[0111] Based on this, embodiments of this application also provide a method for training an object detection model. See [link to relevant documentation]. Figure 3 The diagram shown is a flowchart of a method for training an object detection model according to an embodiment of this application.

[0112] The method includes the following steps: S301: Use an object detection model to perform object detection on multiple autonomous driving images, and determine the loss value and confidence level for each autonomous driving image; S302: Filter multiple autonomous driving images based on loss values ​​and / or confidence levels to determine the number of images to be processed; S303: Determine whether there is an image to be processed. If yes, proceed to step S304; otherwise, proceed to step S310. S304: Determine the target image that matches the image to be processed from the preset image database; S305: Obtain the target mask of the target image; S306: Adjust the brightness of the target image to obtain a brightness-adjusted image; perform shadow processing on the image to be processed based on the target image to obtain a processed image; determine the opacity coefficient of the target image; S307: Image fusion is performed based on the target mask, opacity coefficient, processed image and brightness adjustment image to obtain a fused image; S308: Add the fused image to the training samples and continue training the target detection model using the new training samples; S309: Determine whether the accuracy of the optimized target detection model meets the requirements; if yes, proceed to step S310; otherwise, return to step S301.

[0113] S310: End the training process and output the object detection model.

[0114] The implementation process of the specific steps in the above embodiments can be found in [reference needed]. Figure 1 The embodiments shown will not be described in detail here.

[0115] Based on the above method embodiments, this application also provides a data augmentation device for a target detection model. See also Figure 4 The diagram shown is a schematic of a data augmentation device for a target detection model provided in an embodiment of this application.

[0116] The device 400 includes: The determining unit 401 is used to determine multiple images to be processed based on the target detection model; Matching unit 402 is used to determine a target image that matches the image to be processed in a preset image database, wherein the image database includes multiple preset obstacle images; Acquisition unit 403 is used to acquire the target mask of the target image; The data enhancement unit 404 is used to perform image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image.

[0117] In one possible implementation, the matching unit 402 is configured to: acquire first semantic information of the image to be processed, the first semantic information including at least one of the category of the target in the image to be processed, the illumination information of the image to be processed, and the shooting angle of the image to be processed; acquire second semantic information of the preset obstacle image, the second semantic information including at least one of the category of the preset obstacle in the preset obstacle image, the illumination information of the preset obstacle image, and the shooting angle of the preset obstacle image; determine, among multiple pieces of second semantic information, target semantic information that matches the first semantic information; and determine the preset obstacle image corresponding to the target semantic information as the target image.

[0118] In one possible implementation, when the image to be processed is an image captured by a fisheye camera, the data enhancement unit 404 is used to obtain the fusion position of the target image in the image to be processed; determine the orientation angle corresponding to the fusion position based on the fusion position and the parameters of the fisheye camera; perform fisheye distortion processing on the target image based on the orientation angle to obtain a distorted image; and perform image fusion based on the target mask, the image to be processed, and the distorted image to obtain the fused image.

[0119] In one possible implementation, the data enhancement unit 404 is configured to calculate a first average brightness value of the image to be processed and a second average brightness value of the target image, respectively; adjust the pixel brightness in the target image using the first average brightness value and the second average brightness value to obtain a brightness-adjusted image; and perform image fusion based on the target mask, the image to be processed, and the brightness-adjusted image to obtain the fused image.

[0120] In one possible implementation, the data enhancement unit 404 is configured to determine the light source direction of the image to be processed; perform an affine transformation on the target mask based on the light source direction to determine a shadow mask; process the corresponding region of the image to be processed using the shadow mask to obtain a processed image; and perform image fusion based on the target mask, the processed image, and the target image to obtain the fused image.

[0121] In one possible implementation, the data augmentation unit 404 is used to obtain the physical distance between a preset obstacle in the target image and the camera; determine the opacity coefficient based on the negative correlation between the physical distance and the opacity coefficient; and perform image fusion based on the opacity coefficient, the target mask, the processed image, and the target image to obtain the fused image.

[0122] In one possible implementation, the determining unit 401 is configured to perform target detection on multiple autonomous driving images using the target detection model, obtain the prediction result and confidence level of the predicted target in each autonomous driving image, wherein the target in the autonomous driving image has a real label; determine a loss value based on the prediction result and the real label; and filter the multiple autonomous driving images based on the loss value and / or the confidence level to determine multiple images to be processed.

[0123] In one possible implementation, the acquisition unit 403 is used to perform instance segmentation on the target image to obtain a binary mask of the target image; and to perform Gaussian blur processing on the binary mask to obtain the target mask.

[0124] Based on the above method and device embodiments, this application also provides an electronic device. The following description will be provided in conjunction with the accompanying drawings.

[0125] See Figure 5 , Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application.

[0126] The device 500 includes: a memory 501 and a processor 502; The memory 501 is used to store relevant program code; The processor 502 is used to call the program code and execute the data augmentation method for the target detection model described in the above method embodiments.

[0127] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program for executing the data augmentation method for a target detection model described in the above method embodiments.

[0128] This application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the data augmentation method for the target detection model described in the above method embodiments.

[0129] It should be noted that the computer-readable medium described above in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0130] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0131] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units or modules described as separate components may or may not be physically separate. The components shown as units or modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the units or modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented by methods, apparatuses, and devices according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0134] It should also be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this application can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0136] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data augmentation method for a target detection model, characterized in that, The method includes: Multiple images to be processed were identified based on an object detection model; In a preset image database, a target image that matches the image to be processed is determined. The image database includes multiple preset obstacle images. Obtain the target mask of the target image; Image fusion is performed based on the target mask, the image to be processed, and the target image to obtain a fused image.

2. The method according to claim 1, characterized in that, The step of determining a target image that matches the image to be processed in a preset image database includes: Obtain first semantic information of the image to be processed, the first semantic information including at least one of the following: the category of the target in the image to be processed, the illumination information of the image to be processed, and the shooting angle of the image to be processed; The second semantic information of the preset obstacle image is obtained, and the second semantic information includes at least one of the following: the category of the preset obstacle in the preset obstacle image, the lighting information of the preset obstacle image, and the shooting angle of the preset obstacle image; Among the multiple pieces of second semantic information, determine the target semantic information that matches the first semantic information; The preset obstacle image corresponding to the target semantic information is determined as the target image.

3. The method according to claim 1, characterized in that, When the image to be processed is an image captured by a fisheye camera, the step of image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Obtain the fusion position of the target image in the image to be processed; Based on the fusion location and the parameters of the fisheye camera, determine the orientation angle corresponding to the fusion location; Based on the orientation angle, fisheye distortion processing is performed on the target image to obtain a distorted image; The image is obtained by fusing the target mask, the image to be processed, and the distorted image.

4. The method according to claim 1, characterized in that, The step of fusing images based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Calculate the first average brightness value of the image to be processed and the second average brightness value of the target image, respectively; The brightness of pixels in the target image is adjusted using the first average brightness value and the second average brightness value to obtain a brightness-adjusted image; The image is obtained by fusing the target mask, the image to be processed, and the brightness adjustment image.

5. The method according to claim 1, characterized in that, The step of fusing images based on the target mask, the image to be processed, and the target image to obtain a fused image includes: Determine the direction of the light source in the image to be processed; Affine transformation is performed on the target mask based on the light source direction to determine the shadow mask; The shadow mask is used to process the corresponding region of the image to be processed to obtain the processed image; The image is obtained by fusing the target mask, the processed image, and the target image.

6. The method according to claim 5, characterized in that, The step of fusing images based on the target mask, the processed image, and the target image to obtain the fused image includes: Obtain the physical distance between the preset obstacle in the target image and the camera; The opacity coefficient is determined based on the negative correlation between the physical distance and the opacity coefficient. The image is obtained by fusing the opacity coefficient, the target mask, the processed image, and the target image.

7. The method according to claim 1, characterized in that, The process of determining multiple images to be processed based on the object detection model includes: The target detection model is used to detect targets in multiple autonomous driving images, and the prediction results and confidence scores of the predicted targets in each autonomous driving image are obtained, wherein the targets in the autonomous driving images have real labels; Based on the prediction results and the true labels, the loss value is determined; Based on the loss value and / or the confidence level, multiple autonomous driving images are filtered to determine multiple images to be processed.

8. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the target mask of the target image includes: Instance segmentation is performed on the target image to obtain a binary mask of the target image; The binary mask is subjected to Gaussian blurring to obtain the target mask.

9. A data augmentation device for a target detection model, characterized in that, The device includes: The determination unit is used to determine multiple images to be processed based on the object detection model; A matching unit is used to determine a target image that matches the image to be processed in a preset image database, wherein the image database includes multiple preset obstacle images; The acquisition unit is used to acquire the target mask of the target image; The data enhancement unit is used to perform image fusion based on the target mask, the image to be processed, and the target image to obtain a fused image.

10. An electronic device, characterized in that, The device includes: a memory and a processor; The memory is used to store the relevant program code; The processor is used to call the program code to execute the data augmentation method for the target detection model according to any one of claims 1 to 8.

Images