A data enhancement method and device based on monocular 3D target detection

Abstract

The application discloses a data enhancement method and device based on monocular 3D target detection, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: receiving inputted multiple original images, performing random clipping and scaling processing on each original image to obtain multiple new images; for each new image, based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image, an equivalent camera intrinsic parameter matrix is re-determined; the multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image are inputted into a monocular 3D target detection model for training, so as to update the model parameters and obtain a trained monocular 3D target detection model. The implementation manner uses a random clipping and scaling strategy, solves the problem of uneven distribution of target sample scales, and can greatly improve the detection performance of near obstacle targets.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving in computer technology, and in particular to a data enhancement method and device based on monocular 3D target detection. Background Art

[0002] Monocular 3D object detection is an image-based detection method used for autonomous driving perception tasks. It identifies foreground objects in 2D visual images and determines their category, location, and pose. Compared to lidar and ultrasonic recognition technologies, monocular 3D object detection offers lower sensor costs for environmental perception, but it also suffers from unreliable depth perception.

[0003] Data augmentation is one of the most effective methods for improving model detection performance. To improve the performance of monocular 3D object detection, data augmentation can be performed on the original image. However, due to the constraints of the physical spatial geometric projection relationships of 3D perception, many effective data augmentation methods in 2D, such as rotation, deformation, random cropping, and random scaling, cannot be used in monocular 3D object detection.

[0004] Most monocular 3D object detection methods, such as Mono3D, SMOKE, and Mono3D++, can only use a simple data augmentation strategy called random flipping. However, such methods cannot enhance samples of targets at multiple scales. The imbalance of samples within the scale range is still serious, which will severely limit the detection performance of monocular 3D object detection models for targets of different scales. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a data enhancement method and device based on monocular 3D target detection, which can at least solve the problem that existing monocular 3D target detection cannot achieve multi-scale enhancement of target samples.

[0006] To achieve the above objectives, according to one aspect of an embodiment of the present invention, a data enhancement method based on monocular 3D object detection is provided, comprising:

[0007] Receive multiple original images as input, randomly crop and scale each original image to obtain multiple new images;

[0008] For each new image, the equivalent camera intrinsic parameter matrix is ​​re-determined based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0009] Multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image are input into the monocular 3D object detection model for training to update the model parameters and obtain the trained monocular 3D object detection model.

[0010] Optionally, the randomly cropping and scaling each original image to obtain multiple new images includes:

[0011] Using the cutout height and cutout width, a random cutout operation is performed in the original image to obtain a cutout area; wherein the cutout area does not exceed the boundary of the original image;

[0012] According to the preset image height and preset image width of the monocular 3D object detection model, the cutout area is resized to obtain multiple new images.

[0013] Optionally, before using the cutout height and the cutout width, the method further includes:

[0014] receiving a value selected from a scale factor range, and using the value as a scale factor for cutting out the image;

[0015] The product of the preset image height and the scale factor is used as the cutout height, and the product of the preset image width and the scale factor is used as the cutout width.

[0016] Optionally, re-determining the equivalent camera intrinsic parameter matrix based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image includes:

[0017] Performing a mathematical description of random cropping and scaling with equal probability on the coordinates of each pixel in the cutout area, converting the mathematical description into a determinant, and obtaining an affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0018] The affine transformation relationship is substituted into the projection relationship between the pixel coordinate system of the original image and the camera coordinate system to re-establish the projection relationship between the pixel coordinate system and the camera coordinate system for each pixel in the new image, and an equivalent camera intrinsic parameter matrix is ​​obtained from the new projection relationship.

[0019] Optionally, the mathematical description of performing random cropping and scaling with equal probability on the coordinates of each pixel in the cutout area includes:

[0020] The reciprocal of the scale factor is used as the scaling factor for scaling the size of the cutout area;

[0021] Determine the first coordinates of the upper left corner of each cutout region in the original image and the second coordinates of each pixel in the cutout region in the original image, and subtract the first coordinates from the second coordinates to obtain the coordinates of each pixel in the cutout region;

[0022] The scaling factor is used to scale the coordinates of each pixel in the cutout area to obtain the pixel coordinates of each pixel in the new image.

[0023] Optionally, re-determining the equivalent camera intrinsic parameter matrix based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image includes:

[0024] The reciprocal of the scale factor is used as the scaling factor for scaling the cutout area, and combined with the coordinates of the upper left corner vertex of each cutout area in the original image to calculate the equivalent camera intrinsic parameter matrix.

[0025] Optionally, the target in the original image contains annotated 3D information;

[0026] The method further comprises:

[0027] Obtain a preset reference focal length value for focal length normalization, and use a monocular 3D object detection model to calculate a reference depth value of the object in each new image at the reference focal length value;

[0028] Determining a normalized focal length corresponding to each new image, calculating a ratio of each normalized focal length to the reference focal length value, and obtaining a focal length normalization coefficient corresponding to each new image;

[0029] Multiply each reference depth value by the corresponding focal length normalization coefficient to obtain the inferred depth value of the target in each new image; the inferred depth value is used to compare with the true depth value of the target annotation during the training process of the monocular 3D object detection model to calculate the loss cost.

[0030] To achieve the above object, according to another aspect of an embodiment of the present invention, a data enhancement device based on monocular 3D object detection is provided, comprising:

[0031] The cropping and scaling module is used to receive multiple original images as input and perform random cropping and scaling on each original image to obtain multiple new images;

[0032] A matrix determination module is used to re-determine the equivalent camera intrinsic parameter matrix for each new image based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0033] The model training module is used to input multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image into the monocular 3D object detection model for training to update the model parameters and obtain a trained monocular 3D object detection model.

[0034] Optionally, the cropping and scaling module is used to:

[0035] Using the cutout height and cutout width, a random cutout operation is performed in the original image to obtain a cutout area; wherein the cutout area does not exceed the boundary of the original image;

[0036] According to the preset image height and preset image width of the monocular 3D object detection model, the cutout area is resized to obtain multiple new images.

[0037] Optionally, the cropping and scaling module is further configured to:

[0038] receiving a value selected from a scale factor range, and using the value as a scale factor for cutting out the image;

[0039] The product of the preset image height and the scale factor is used as the cutout height, and the product of the preset image width and the scale factor is used as the cutout width.

[0040] Optionally, the matrix determination module is used to:

[0041] Performing a mathematical description of random cropping and scaling with equal probability on the coordinates of each pixel in the cutout area, converting the mathematical description into a determinant, and obtaining an affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0042] The affine transformation relationship is substituted into the projection relationship between the pixel coordinate system of the original image and the camera coordinate system to re-establish the projection relationship between the pixel coordinate system and the camera coordinate system for each pixel in the new image, and an equivalent camera intrinsic parameter matrix is ​​obtained from the new projection relationship.

[0043] Optionally, the matrix determination module is used to:

[0044] The reciprocal of the scale factor is used as a scaling factor for scaling the cutout area;

[0045] Determine the first coordinates of the upper left corner of each cutout region in the original image and the second coordinates of each pixel in the cutout region in the original image, and subtract the first coordinates from the second coordinates to obtain the coordinates of each pixel in the cutout region;

[0046] The scaling factor is used to scale the coordinates of each pixel in the cutout area to obtain the pixel coordinates of each pixel in the new image.

[0047] Optionally, the matrix determination module is used to:

[0048] The reciprocal of the scale factor is used as the scaling factor for scaling the cutout area, and combined with the coordinates of the upper left corner vertex of each cutout area in the original image to calculate the equivalent camera intrinsic parameter matrix.

[0049] Optionally, the target in the original image contains annotated 3D information;

[0050] The device further includes a focal length normalization module, configured to:

[0051] Obtain a preset reference focal length value for focal length normalization, and use a monocular 3D object detection model to calculate a reference depth value of the object in each new image at the reference focal length value;

[0052] Determining a normalized focal length corresponding to each new image, calculating a ratio of each normalized focal length to the reference focal length value, and obtaining a focal length normalization coefficient corresponding to each new image;

[0053] Multiply each reference depth value by the corresponding focal length normalization coefficient to obtain the inferred depth value of the target in each new image; the inferred depth value is used to compare with the true depth value of the target annotation during the training process of the monocular 3D object detection model to calculate the loss cost.

[0054] To achieve the above objectives, according to another aspect of an embodiment of the present invention, a data enhancement electronic device based on monocular 3D target detection is provided.

[0055] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data enhancement methods based on monocular 3D target detection.

[0056] To achieve the above objectives, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements any of the above-mentioned data enhancement methods based on monocular 3D target detection.

[0057] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: the monocular 3D target detection model requires two inputs: image data and the camera intrinsic parameter matrix. This solution performs random cropping and scaling processing in the image input stage to achieve data enhancement, and adopts the focal length normalization method. In the depth decoding stage after the 3D box regression of the network model, the focal length normalization coefficient is introduced to solve the depth reasoning problem of images with different focal lengths, thereby improving the detection performance of the subsequent monocular 3D target detection model as a whole.

[0058] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0060] Figure 1 1 is a schematic diagram of the main process of a data enhancement method based on monocular 3D object detection according to an embodiment of the present invention;

[0061] Figure 2 1 is a flow chart of a method for performing random_crop_resize on an original image according to an embodiment of the present invention;

[0062] Figure 3 3D object detection is an optional data enhancement method based on monocular 3D object detection according to an embodiment of the present invention;

[0063] Figure 4 3D object detection based on a monocular 3D object detection apparatus according to an embodiment of the present invention;

[0064] Figure 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0065] Figure 6 It is a schematic diagram of the structure of a computer system of a mobile device or server suitable for implementing the embodiments of the present invention. DETAILED DESCRIPTION

[0066] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0067] Data augmentation is one of the most effective methods for improving model detection performance without incurring additional computational cost during inference. Random scaling, random cropping, color distortion, and other geometry and color augmentation techniques are widely used in 2D detection models. In addition to common data augmentation techniques, copy-paste augmentation is also widely used in 2D detection and segmentation tasks.

[0068] However, due to geometric constraints, no monocular 3D object detection method uses these data augmentation methods, leaving horizontal flipping and color distortion as the only data augmentation methods. Monocular original images are derived from perspective imaging, and the scale distribution of objects in monocular images is severely imbalanced, limiting the effectiveness of model training. This method is unable to enhance multi-scale samples of objects, and the scale imbalance of samples remains severe, severely limiting the performance of monocular 3D object detection models for objects of varying sizes.

[0069] See also Figure 1 , which shows a main flow chart of a data enhancement method based on monocular 3D target detection provided by an embodiment of the present invention, including the following steps:

[0070] S101: receiving multiple original images as input, and performing random cropping and scaling processing on each original image to obtain multiple new images;

[0071] S102: For each new image, re-determine the equivalent camera intrinsic parameter matrix based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0072] S103: Input the multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image into the monocular 3D object detection model for training to update the model parameters and obtain a trained monocular 3D object detection model.

[0073] In the above implementation, for step S101, monocular 3D target detection is mainly applicable to the obstacle 3D target detection scenario in the field of autonomous driving. This solution is applied to the data preprocessing scenario in the monocular 3D target detection model training stage.

[0074] Monocular cameras can only capture two-dimensional images. Existing monocular 3D object detection methods, such as SMOKE, take the original image and the camera's intrinsic parameter matrix as input. They detect obstacles (such as vehicles and people) in the camera coordinate system, including their physical position (x, y, z), physical dimensions (length, width, height) (w, h, l), and attitude angle (yaw), forming the 3D information (x, y, z, w, h, l, yaw). SMOKE is a one-stage monocular 3D object detection algorithm based on keypoint detection. Therefore, this solution annotates objects in the original image with 3D information.

[0075] Currently, the original images are typically fed into a monocular 3D object detection model for training according to the batch_size. For example, a total of 10,000 original images are divided into 10 batches, with only 1,000 original images fed into each batch. This results in the possibility that the test images may be identical to the training images, or that images from different batches may be identical. Furthermore, each original image only has one scale for obstacles, making it difficult to effectively train the monocular 3D object detection model.

[0076] The monocular 3D target detection model is based on the original image 3D target detection. To solve the above problem, this solution adopts the random_crop_resize (random cropping and scaling) method for the original image. For details, see Figure 2 As shown:

[0077] S201: receiving a value selected from a scale coefficient range, and using the value as a scale coefficient for cutting out the image;

[0078] S202: Obtaining a preset image height and a preset image width of a monocular 3D object detection model, and using the product of the preset image height and the scale coefficient as a cutout height, and the product of the preset image width and the scale coefficient as a cutout width;

[0079] S203: Using the cutout height and cutout width, perform a cutout operation at a random position in each original image to obtain a cutout area; wherein the cutout area does not exceed the boundary of the original image;

[0080] S204: scaling the cutout area according to a preset image height and a preset image width to obtain a plurality of new images.

[0081] 1) Set the minimum scale min_scale (e.g., 0.7) and maximum scale max_scale (e.g., 1.0) for the cutout, and randomly select a value from the scale coefficient range [min_scale, max_scale) as the scale coefficient for the cutout. This process can be done manually or by programming the operator to select the scale coefficient using equal probability. Considering the automated operation of this solution, programming the scale coefficient using equal probability is preferred.

[0082] 2) The image width and image height of the monocular 3D object detection model are preset to infer_w and infer_h respectively, so the calculated cutout width is crop_w = scale * infer_w, and the cutout height is crop_h = scale * infer_h.

[0083] 3) Based on crop_w and crop_h, a random cropping operation is performed in the original input single-frame original image, and the cropped area does not exceed the original image area. Therefore, the cropped image does not have the problem of zero padding due to exceeding the original image boundary. This makes the model training phase and the data preprocessing phase consistent, which helps to improve the performance of the model deployment phase.

[0084] 4) The cropped image is resized to crop_w and crop_h. A new image suitable for the monocular 3D object detection model is generated with the width and height of the new image being infer_w and infer_h respectively.

[0085] For step S102, it is assumed that initially, the projection relationship between the pixel coordinate system of the original image and the camera coordinate system is as follows:

[0086]

[0087] Among them, u and v are the horizontal coordinates and vertical coordinates (i.e., the second coordinates) of a pixel on the original image in the pixel coordinate system, K is the camera intrinsic parameter matrix, Xc 、Y c and Z c is the coordinate of the pixel in the camera coordinate system, that is, the position information.

[0088] After the original image is processed by random_crop_resize to obtain multiple new images, the coordinates of each pixel in the original image have changed. As a result, if the pixels in the 3D space of the camera coordinate system continue to be projected onto the new image using the original intrinsic projection matrix, there will be a coordinate mismatch problem. That is, the coordinates of the pixels in the new image can no longer form a perspective geometric relationship in the original pixel coordinate system and the camera coordinate system. For this reason, this solution re-establishes the projection relationship between the new pixel coordinate system and the camera coordinate system through theoretical derivation. The following is a detailed explanation:

[0089] 1. For the coordinates of each pixel in the cutout area, perform mathematical description of random cropping and scaling with equal probability:

[0090] u'=s(u-u1)

[0091] v'=s(v-v1)

[0092] Among them, u' is the horizontal coordinate of the pixel in the new image in the pixel coordinate system, v' is the vertical coordinate of the pixel in the new image in the pixel coordinate system; s is the scaling factor, which is the reciprocal of the aforementioned scale factor scale; u1 and v1 are the first coordinates of the upper left corner vertex of the cutout area in the original image, and the original image has the upper left corner vertex as the origin.

[0093] Converting these two formulas into determinant form, we can derive the affine transformation relationship between the coordinates of each pixel in the original image and the new image:

[0094]

[0095] 2. Substitute the above "affine transformation relationship between the coordinates of each pixel in the original image and the new image" into the "projection relationship between the pixel coordinate system of the original image and the camera coordinate system" to re-derive the "perspective projection relationship between the pixel coordinate system of each pixel in the new image and the camera coordinate system".

[0096]

[0097]

[0098] Among them, K' is the equivalent camera intrinsic parameter matrix corresponding to the new image. When crop_resize is performed on the original image, it is equivalent to performing an affine transformation on the pixel coordinates. The pixel coordinate system of the pixels in the new image, the new equivalent intrinsic parameter matrix, and the camera coordinate system can re-establish the geometric perspective relationship.

[0099] In step S103, the augmented new images and the equivalent camera intrinsic parameter matrix corresponding to each new image are input into the monocular 3D object detection model for training to update the model parameters. The trained monocular 3D object detection model can correctly infer the 3D information (x, y, z, w, h, l, yaw) of obstacles and targets from a single frame during the inference phase.

[0100] Here is a specific example:

[0101] Assume that the pixels of the input original image are 640*480, and the original camera intrinsic parameter matrix K is:

[0102]

[0103] The selected cropping scale factor scale = 0.8, so the size of the new image 512*384 is smaller than the size of the original image. The parameters obtained by random_crop_resize include: scaling factor s = 1 / scale = 1.25, coordinates of the upper left corner of the new image in the original image u1 = 24, v1 = 32;

[0104] Input the above parameters into the new camera intrinsic parameter matrix formula to obtain K':

[0105]

[0106] Then, the geometric perspective relationship of the new image is re-established based on the above K' to achieve data enhancement of the pixel scale and pixel position of the pixel points. That is, the generated new image is equivalent to the original image taken at the same position with a new camera.

[0107] The method provided in the above embodiment is mainly aimed at data preprocessing scenarios during model training. By applying a random_crop_resize strategy to the input original image without exceeding the image boundary, multiple new images with the obstacle target position offset and resized can be generated, and the size interval distribution in these new images is balanced. At the same time, the equivalent camera intrinsic parameter matrix of the new image is updated according to the above K' to re-establish the projection relationship.

[0108] See also Figure 3 , shows a schematic flow chart of an optional data enhancement method based on monocular 3D target detection according to an embodiment of the present invention, comprising the following steps:

[0109] S301: Obtaining a reference focal length value preset for focal length normalization, and calculating a reference depth value of a target in each new image at the reference focal length value using a monocular 3D target detection model; wherein the target in the original image includes annotated 3D information;

[0110] S302: Determine a normalized focal length corresponding to each new image, calculate a ratio of each normalized focal length to the reference focal length value, and obtain a focal length normalization coefficient corresponding to each new image;

[0111] S303: Multiply each reference depth value by the corresponding focal length normalization coefficient to obtain the inferred depth value of the target in each new image; wherein the inferred depth value is used for the monocular 3D target detection model during the training process to compare with the true depth value of the target annotation to calculate the loss cost.

[0112] In the above implementation, for steps S301 to S303, the original image input by this solution already has 3D information (x, y, z, w, h, l, yaw) annotated for the obstacle target. Perspective imaging is characterized by objects appearing larger near and smaller far away. The pixel height of the target in the original image actually reflects the target's depth value z. Therefore, under the supervision of this information (the target's 3D information), the model can infer the target's depth value z and other 3D information.

[0113] Some new images have different focal lengths, for example, image 1 has a focal length of 6mm and image 2 has a focal length of 12mm. When these images are mixed for training, the actual depth values ​​of objects with the same pixel height in images with different focal lengths will be different, so focal length normalization is required using the ratio formula based on perspective principle:

[0114]

[0115] Where n is the number of pixels of the predefined target in the vertical height direction, d y is the pixel size of the camera sensor, h is the height of the target, and d is the distance from the target to the camera, that is, the depth value depth.

[0116] By transforming the above formula, we can get the new formula:

[0117]

[0118] Among them, f ref A constant reference focal length value introduced for focal length normalization, such as f ref =720.0;f y is the normalized focal length in the camera intrinsic parameter matrix K, f y / f ref is the focal length normalization coefficient, which is different for different images; F regress For the SMOKE algorithm model, when the pixel height of the obstacle target in the image is n, the reference focal length value f ref The reference depth value obtained by the regression process.

[0119] The inferred depth value of the target in each new image is the focal length factor f y / f ref Multiply by F regress ,The inferred depth value is used in the monocular 3D object detection model during ,training process and is compared with the true depth value of the target annotation to ,calculate the loss cost.

[0120] The method provided in the above embodiment, during the training process of obtaining multiple new images through data augmentation, leads to the technical problem of different image focal lengths. This solution adopts the focal length normalization method to solve the training and inference problem of images with different focal lengths, and avoids the depth value regression problem caused by different pixel heights of the same target under cameras with different focal lengths.

[0121] The method provided by the embodiment of the present invention has at least the following beneficial effects compared to the prior art:

[0122] 1. A random cropping and scaling strategy is applied to the input original image without exceeding the image boundary. This generates multiple new images with the obstacle target position offset and size scaled, achieving the purpose of sample scale enhancement and greatly alleviating the distribution balance of target samples in the scale range.

[0123] 2. By updating the camera intrinsic parameter matrix, the perspective geometry constraints are still satisfied even when the target sample size is changed, thereby significantly improving the performance of the monocular 3D object detection model;

[0124] 3. The depth estimation method using focal length normalization can effectively solve the problem of inconsistent focal lengths of different new images and the conflict of supervised information during mixed training of images with multiple focal lengths, so that the model deployment supports cameras with different focal lengths.

[0125] See also Figure 4 , which shows a schematic diagram of the main modules of a data enhancement device 400 based on monocular 3D target detection provided by an embodiment of the present invention, including:

[0126] The cropping and scaling module 401 is configured to receive multiple original images as input and perform random cropping and scaling on each original image to obtain multiple new images;

[0127] A matrix determination module 402 is configured to re-determine, for each new image, an equivalent camera intrinsic parameter matrix based on an affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0128] The model training module 403 is used to input multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image into the monocular 3D object detection model for training to update the model parameters and obtain a trained monocular 3D object detection model.

[0129] In the implementation device of the present invention, the cropping and scaling module 401 is used to:

[0130] Using the cutout height and cutout width, a random cutout operation is performed in the original image to obtain a cutout area; wherein the cutout area does not exceed the boundary of the original image;

[0131] According to the preset image height and preset image width of the monocular 3D object detection model, the cutout area is resized to obtain multiple new images.

[0132] In the implementation device of the present invention, the cropping and scaling module 401 is further configured to:

[0133] receiving a value selected from a scale factor range, and using the value as a scale factor for cutting out the image;

[0134] The product of the preset image height and the scale factor is used as the cutout height, and the product of the preset image width and the scale factor is used as the cutout width.

[0135] In the implementation device of the present invention, the matrix determination module 402 is used to:

[0136] Performing a mathematical description of random cropping and scaling with equal probability on the coordinates of each pixel in the cutout area, converting the mathematical description into a determinant, and obtaining an affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0137] The affine transformation relationship is substituted into the projection relationship between the pixel coordinate system of the original image and the camera coordinate system to re-establish the projection relationship between the pixel coordinate system and the camera coordinate system for each pixel in the new image, and an equivalent camera intrinsic parameter matrix is ​​obtained from the new projection relationship.

[0138] In the implementation device of the present invention, the matrix determination module 402 is used to:

[0139] The reciprocal of the scale factor is used as a scaling factor for scaling the cutout area;

[0140] Determine the first coordinates of the upper left corner of each cutout region in the original image and the second coordinates of each pixel in the cutout region in the original image, and subtract the first coordinates from the second coordinates to obtain the coordinates of each pixel in the cutout region;

[0141] The scaling factor is used to scale the coordinates of each pixel in the cutout area to obtain the pixel coordinates of each pixel in the new image.

[0142] In the implementation device of the present invention, the matrix determination module 402 is used to:

[0143] The reciprocal of the scale factor is used as the scaling factor for scaling the cutout area, and combined with the coordinates of the upper left corner vertex of each cutout area in the original image to calculate the equivalent camera intrinsic parameter matrix.

[0144] In the device implementing the present invention, the target in the original image includes annotated 3D information;

[0145] The device further includes a focal length normalization module, configured to:

[0146] Obtain a preset reference focal length value for focal length normalization, and use a monocular 3D object detection model to calculate a reference depth value of the object in each new image at the reference focal length value;

[0147] Determining a normalized focal length corresponding to each new image, calculating a ratio of each normalized focal length to the reference focal length value, and obtaining a focal length normalization coefficient corresponding to each new image;

[0148] Multiply each reference depth value by the corresponding focal length normalization coefficient to obtain the inferred depth value of the target in each new image; the inferred depth value is used to compare with the true depth value of the target annotation during the training process of the monocular 3D object detection model to calculate the loss cost.

[0149] In addition, the specific implementation content of the device in the embodiment of the present invention has been described in detail in the above method, so the repeated content will not be described again here.

[0150] Figure 5 An exemplary system architecture 500 to which embodiments of the present invention may be applied is shown, including terminal devices 501 , 502 , 503 , a network 504 and a server 505 (only an example).

[0151] Terminal devices 501, 502, 503 can be various electronic devices with display screens and support web browsing, and are installed with various communication client applications. Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc.

[0152] The network 504 is used to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0153] The server 505 may be a server that provides various services, and is used to perform operations such as acquiring a new image using a random cropping and scaling strategy, determining an equivalent camera intrinsic parameter matrix for the new image, and using the two to train a monocular 3D target detection model.

[0154] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device is generally set in the server 505 .

[0155] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0156] Reference below Figure 6 , which shows a schematic structural diagram of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0157] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0158] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0159] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are performed.

[0160] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as: a processor including a cropping and scaling module, a matrix determination module, and a model training module. The names of these modules do not, in some cases, limit the modules themselves. For example, the model training module may also be described as a "model monocular 3D object detection model training module."

[0163] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0164] Receive multiple original images as input, randomly crop and scale each original image to obtain multiple new images;

[0165] For each new image, the equivalent camera intrinsic parameter matrix is ​​re-determined based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image;

[0166] Multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image are input into the monocular 3D object detection model for training to update the model parameters and obtain the trained monocular 3D object detection model.

[0167] The technical solution according to the embodiments of the present invention has at least the following beneficial effects compared to the prior art:

[0168] 1. A random cropping and scaling strategy is applied to the input original image without exceeding the image boundary. This generates multiple new images with the obstacle target position offset and size scaled, achieving the purpose of sample scale enhancement and greatly alleviating the distribution balance of target samples in the scale range.

[0169] 2. By updating the camera intrinsic parameter matrix, the perspective geometry constraints are still satisfied even when the target sample size is changed, thereby significantly improving the performance of the monocular 3D object detection model;

[0170] 3. The depth estimation method using focal length normalization can effectively solve the problem of inconsistent focal lengths of different new images and the conflict of supervised information during mixed training of images with multiple focal lengths, so that the model deployment supports cameras with different focal lengths.

[0171] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data enhancement method based on monocular 3D target detection, characterized in that: include: Receive multiple original images as input, randomly crop and scale each original image to obtain multiple new images; wherein the objects in the original images contain annotated 3D information; For each new image, the equivalent camera intrinsic parameter matrix is ​​re-determined based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image; Multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image are input together into a monocular 3D target detection model for training to update the model parameters and obtain a trained monocular 3D target detection model; wherein, a reference focal length value preset for focal length normalization is obtained, and the monocular 3D target detection model is used to calculate the reference depth value of the target in each new image at the reference focal length value; the normalized focal length corresponding to each new image is determined, and the ratio of each normalized focal length to the reference focal length value is calculated to obtain a focal length normalization coefficient corresponding to each new image; each reference depth value is multiplied by the corresponding focal length normalization coefficient to obtain an inferred depth value of the target in each new image; wherein, the inferred depth value is used for comparison with the true depth value of the target annotation during the training process of the monocular 3D target detection model to calculate the loss cost.

2. The method according to claim 1, characterized in that The random cropping and scaling process is performed on each original image to obtain multiple new images, including: Using the cutout height and cutout width, a random cutout operation is performed in each original image to obtain a cutout area; wherein the cutout area does not exceed the boundary of the original image; According to the preset image height and preset image width of the monocular 3D object detection model, the cutout area is resized to obtain multiple new images.

3. The method according to claim 2, characterized in that Before using the cutout height and cutout width, it also includes: receiving a value selected from a scale factor range, and using the value as a scale factor for cutting out the image; The product of the preset image height and the scale factor is used as the cutout height, and the product of the preset image width and the scale factor is used as the cutout width.

4. The method according to claim 3, characterized in that The method of re-determining the equivalent camera intrinsic parameter matrix based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image includes: Performing a mathematical description of random cropping and scaling with equal probability on the coordinates of each pixel in the cutout area, converting the mathematical description into a determinant, and obtaining an affine transformation relationship between the coordinates of each pixel in the original image and the new image; The affine transformation relationship is substituted into the projection relationship between the pixel coordinate system of the original image and the camera coordinate system to re-establish the projection relationship between the pixel coordinate system and the camera coordinate system for each pixel in the new image, and an equivalent camera intrinsic parameter matrix is ​​obtained from the new projection relationship.

5. The method according to claim 4, characterized in that The mathematical description of performing random clipping and scaling with equal probability on the coordinates of each pixel in the cutout area includes: The reciprocal of the scale factor is used as a scaling factor for scaling the cutout area; Determine the first coordinates of the upper left corner of the cutout area in the original image and the second coordinates of each pixel in the cutout area in the original image, and subtract the first coordinates from the second coordinates to obtain the coordinates of each pixel in the cutout area; The coordinates of each pixel in the cutout area are scaled using the scaling factor to obtain the coordinates of each pixel in the new image.

6. The method according to claim 3, characterized in that The method of re-determining the equivalent camera intrinsic parameter matrix based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image includes: The reciprocal of the scale factor is used as the scaling factor for scaling the cutout area, and combined with the coordinates of the upper left corner vertex of each cutout area in the original image to calculate the equivalent camera intrinsic parameter matrix.

7. A data enhancement device based on monocular 3D target detection, characterized in that: include: The cropping and scaling module is used to receive multiple original images as input and randomly crop and scale each original image to obtain multiple new images; wherein the objects in the original images contain annotated 3D information; A matrix determination module is used to re-determine the equivalent camera intrinsic parameter matrix for each new image based on the affine transformation relationship between the coordinates of each pixel in the original image and the new image; The model training module is used to input multiple new images and the equivalent camera intrinsic parameter matrix corresponding to each new image into the monocular 3D target detection model for training to update the model parameters and obtain the trained monocular 3D target detection model; wherein, a reference focal length value preset for focal length normalization is obtained, and the monocular 3D target detection model is used to calculate the reference depth value of the target in each new image at the reference focal length value; determine the normalized focal length corresponding to each new image, calculate the ratio of each normalized focal length to the reference focal length value, and obtain the focal length normalization coefficient corresponding to each new image; multiply each reference depth value by the corresponding focal length normalization coefficient to obtain the inferred depth value of the target in each new image; wherein, the inferred depth value is used for the monocular 3D target detection model to compare with the true depth value of the target annotation during the training process to calculate the loss cost.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images