Data Augmentation for Object Detection via Differential Neural Rendering

Abstract

A system and method for detecting objects using an enhanced training data set. The system includes a computing device, the computing device is used to: provide a two-dimensional (2D) image and a projector position, extract feature vectors of 2D image pixels and estimate the depth of the 2D image pixels, generate a point cloud using the feature vectors and pixel depth; and project the point cloud using a new projector position to obtain a projected image. The 2D image includes a border, the border surrounds the object, and is annotated according to the object. Each pixel within the border is called a border pixel, each point in the point cloud corresponding to the border pixel is called a border point, each image pixel in the projected image corresponding to the border point is called a projection border pixel, the projection border pixels are used to define the projection border, and the projection border is annotated with the object.

Description

Technical Field

[0001] The present disclosure relates to object detection training using augmented data, and more particularly, to generating augmented data using differential neural rendering. Background Art

[0002] The background description provided herein is intended to present the context of the present disclosure as a whole. The scope described in the background section and the aspects described herein that may not constitute prior art when filing an application for the work of the current inventor do not constitute prior art of the present disclosure, either explicitly or implicitly.

[0003] Object detection neural networks or object detectors require the use of annotated data for training. When annotated data is scarce, training object detectors is challenging because the detector needs to handle objects that appear at arbitrary locations in the image and therefore requires a large amount of data. Further, traditional online data augmentation methods include random cropping and image mixing. Online data augmentation methods provide additional data diversity to avoid overfitting, but do not provide invisible semantics and new locations. Moreover, as an offline data augmentation method, training images can be synthesized. However, simply pasting the object foreground to a new background (alpha blending) is not natural because it introduces artificial traces that may damage or destroy the detection model. In addition, it also requires additional labeling, that is, annotating the object foreground at the pixel level.

[0004] Therefore, there is an unresolved need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the invention

[0005] In some aspects, the present disclosure relates to a system. In some embodiments, the system includes a computing device, the computing device including a processor and a storage device storing computer executable code, wherein the computer executable code when executed at the processor is used to:

[0006] Providing a two-dimensional (2D) image and a projector position, wherein the 2D image includes a frame, the frame surrounds the object, and the frame is labeled according to the object;

[0007] Extract feature vectors of pixels in the 2D image, each pixel having a corresponding feature vector;

[0008] Estimate pixel depth from 2D images;

[0009] Generate a point cloud using the feature vectors and pixel depths; and

[0010] Use a projector to project the point cloud to obtain a projection image.

[0011] Among them, each pixel within the border has a border label and is called a border pixel, each point in the point cloud corresponding to the border pixel has a border label and is called a border point, each image pixel corresponding to the border point in the projected image has a border label and is called a projection border pixel, and the projection border pixels are used to define the projection border.

[0012] In some embodiments, the computer executable code is used to extract feature points using an encoder with an Inception-ResNet block.

[0013] In some embodiments, the computer executable code is used to estimate depth using an hourglass network.

[0014] In some embodiments, when the projection position is the original position of the camera that captures the 2D image, the computer executable code is used to project the point cloud using the following formula:

[0015]

[0016] Among them, (u, v) is the coordinate of the target point in the point cloud, d is the depth of the target point in the point cloud, c x and c y is the offset of the center of the camera, f x and f y is the focal length of the camera on the x-axis and y-axis, and (X c , Y c , Z c ) are the coordinates of the target point in the camera coordinate system.

[0017] In some embodiments, the computer executable code is used to compare the projected image with the 2D image to learn parameters for extracting feature key points, estimating depth, generating a point cloud, and projecting the point cloud.

[0018] In some embodiments, when the projection position is a new position with a rotation R and a translation T relative to the original position, the computer executable code is used to project the point cloud using the following formula:

[0019] P c =R·(P w -T) formula (2), and

[0020]

[0021] Among them, P w Equivalent to formula (1), P c is the coordinate of the target point in the new camera coordinate system corresponding to the new camera position, P c By (X′ c , Y′ c , Z′ c) denotes, and (u′, v′) are the coordinates of the pixel in the projection image corresponding to the target point.

[0022] In some embodiments, the projected bounding box is defined by the following formula:

[0023] Where P′ i ={(u′ i , v′ i )} are the coordinates of the pixels in the projected bounding box.

[0024] In some embodiments, the computer executable code is further configured to:

[0025] Combining the 2D image, bounding box, projected image, and projected bounding box to form a training dataset;

[0026] Train a neural network for object detection using a training dataset;

[0027] Use neural networks to detect objects in images retrieved from commercial platforms;

[0028] When a detection object of one of the retrieved images belongs to a predetermined object type, a warning message is sent with respect to the one of the retrieved images.

[0029] In some embodiments, the projection camera positions include four different camera positions different from the original camera positions for capturing 2D images.

[0030] In some embodiments, the number of the projector positions is four, and the four projector positions are substantially evenly distributed in the three-dimensional space relative to the original position.

[0031] In some aspects, the present disclosure relates to a method. In some embodiments, the method includes:

[0032] Providing a two-dimensional (2D) image and a projector position, wherein the 2D image includes a frame, the frame surrounds the object, and the frame is labeled according to the object;

[0033] Extracting feature vectors of pixels in the 2D image, each pixel including a corresponding feature vector;

[0034] Estimate pixel depth from 2D images;

[0035] Generate a point cloud using the feature vectors and pixel depths; and

[0036] Use a projector to project the point cloud to obtain a projection image.

[0037] Among them, each pixel within the border has a border label and is called a border pixel, each point in the point cloud corresponding to the border pixel has a border label and is called a border point, each image pixel corresponding to the border point in the projected image has a border label and is called a projection border pixel, and the projection border pixels are used to define the projection border.

[0038] In some embodiments, the feature vector is extracted using an encoder with an Inception-ResNet block.

[0039] In some embodiments, depth is estimated using an hourglass network.

[0040] In some embodiments, when the projection position is the original position of the camera that captures the 2D image, the point cloud is projected using the following formula:

[0041]

[0042] Among them, (u, v) is the coordinate of the target point in the point cloud, d is the depth of the target point in the point cloud, c x and c v is the offset of the center of the camera, f x and f y is the focal length of the camera on the x-axis and y-axis, and (X c , Y c , Z c ) are the coordinates of the target point in the camera coordinate system.

[0043] In some embodiments, the method further includes comparing the projected image with the 2D image to learn parameters for extracting feature key points, estimating depth, generating a point cloud, and projecting the point cloud.

[0044] In some embodiments, when the projection position is a new position with a rotation R and a translation T relative to the original position, the point cloud is projected using the following formula:

[0045] P c =R·(P w -T) formula (2), and

[0046]

[0047] Among them, P w Equivalent to formula (1), P c is the coordinate of the target point in the new camera coordinate system corresponding to the new camera position, P c By (X′ c , Y′ c , Z′ c ) denotes, and (u′, v′) are the coordinates of the pixel in the projection image corresponding to the target point.

[0048] In some embodiments, the projected bounding box is defined by the following formula:

[0049] Where P′ i ={(u′ i , v′ i )} are the coordinates of the pixels in the projected bounding box.

[0050] In some embodiments, the method further comprises:

[0051] Combining the 2D image, bounding box, projected image, and projected bounding box to form a training dataset;

[0052] Train a neural network for object detection using a training dataset;

[0053] Use neural networks to detect objects in images retrieved from commercial platforms;

[0054] When a detection object of one of the retrieved images belongs to a predetermined object type, a warning message is sent with respect to the one of the retrieved images.

[0055] In some embodiments, the projection camera positions include four different camera positions different from the original camera positions for capturing 2D images.

[0056] In some aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. The computer executable code is used to perform the above method when executed at a processor of a computing device.

[0057] These and other aspects of the present disclosure will become apparent from the following description of preferred embodiments taken in conjunction with the accompanying drawings and illustrations thereof, and changes and modifications may be made without departing from the spirit and scope of the novel concepts of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which illustrate one or more embodiments of the present disclosure, together with the description, explain the principles of the present disclosure. Wherever possible, in the drawings, the same reference numerals refer to the same or similar elements in the embodiments.

[0059] Figure 1 A data enhancement based on neural rendering (DANR) system according to some embodiments of the present disclosure is schematically illustrated.

[0060] Figure 2 The operation of the new view synthesis model according to some embodiments of the present disclosure is schematically illustrated.

[0061] Figure 3 The implementation of DANR according to some embodiments of the present disclosure is schematically illustrated.

[0062] Figure 4AThe inception-residual block of DANR that preserves feature resolution according to some embodiments of the present disclosure is schematically illustrated.

[0063] Figure 4B The inception-residual block of DANR for improving feature resolution according to some embodiments of the present disclosure is schematically shown.

[0064] Figure 4C The inception-residual block of DANR for reducing feature resolution according to some embodiments of the present disclosure is schematically shown.

[0065] Figure 4D Schematically illustrating a feature encoder stacked from inception-residual blocks according to some embodiments of the present disclosure.

[0066] Figure 4E A decoder according to some embodiments of the present disclosure is schematically shown.

[0067] Figure 4F The architecture of a depth estimator of DANR according to some embodiments of the present disclosure is schematically shown.

[0068] Figure 5A A differentiable renderer according to some embodiments of the present disclosure is schematically shown.

[0069] Figure 5B Four camera positions for generating new images according to some embodiments of the present disclosure are schematically shown.

[0070] Figure 6 The process for training an enhanced model according to some embodiments of the present disclosure is schematically illustrated.

[0071] Figure 7 The following schematically illustrates a process for enhancing a training dataset according to some embodiments of the present disclosure.

[0072] Figure 8 The diagram schematically illustrates a process of training an object detector using an enhanced training dataset according to some embodiments of the present disclosure.

[0073] Fig.9A The performance of DANR using high rendering resolution and low rendering resolution according to some embodiments of the present disclosure is compared.

[0074] Fig. 9B The performance of DANR according to some embodiments of the present disclosure is compared using different training: validation data split ratios.

[0075] Fig. 9C The performance of DANR using different frameworks according to some embodiments of the present disclosure is compared.

[0076] Fig.9D The performance of DANR using primary and secondary detection frameworks according to some embodiments of the present disclosure is compared.

[0077] Fig.9E The performance of DANR using different types of data sets according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0078] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations therein will be apparent to those skilled in the art. Embodiments of the present disclosure are now described in detail. Reference is made to the accompanying drawings, in which the same numbers refer to the same components. Unless the context clearly dictates otherwise, the meanings of "one", "an" and "said" as used in the description herein and in the subsequent claims include the plural, and "multiple" represents two or more. As used herein, the terms "comprising", "including", "carrying", "having", "including", "involving", etc. should be understood as open-ended, i.e., meaning including but not limited to. In addition, titles or subtitles are used in the specification for the convenience of the reader, which does not affect the scope of the present disclosure.

[0079] The terms used in this specification generally have their ordinary meanings in the art, in the context of the present disclosure, and in the specific context in which each term is used. Some terms used to describe the present disclosure are discussed below or elsewhere in this specification to provide additional guidance to practitioners about the present disclosure description content. It is understood that the same thing can be described in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed in this article, and there is no special meaning for whether the term is elaborated or discussed in detail in this article. The one or more synonyms recorded do not exclude the use of other synonyms. The use of any example in this specification, including the example of any term discussed herein, is only exemplary and does not limit the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments provided in this specification.

[0080] As used herein, the phrase “at least one of A, B, and C” should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure.

[0081] The term "module" as used herein may refer to a portion of, or include, an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a (shared, dedicated, or group) processor that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, for example, in a system on a chip. The term "module" may include a (shared, dedicated, or group) memory that stores code executed by a processor.

[0082] The term "code" used herein may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The shared term used above means that a single (shared) processor may be used to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored in a single (shared) memory. The term group used above means that a group of processors may be used to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0083] As used herein, the term "interface" generally refers to a communication tool or device that performs data communication between components at the point of interaction between components. Typically, interfaces are applicable at both the hardware and software levels, and can be unidirectional or bidirectional interfaces. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0084] The present disclosure relates to computer systems. As shown in the figure, computer components may include physical hardware components shown as solid blocks, and virtual software components shown as dashed blocks. It should be understood by those skilled in the art that, unless otherwise specified, these computer components may be, but are not limited to, implemented in the form of software, firmware, hardware components, or a combination thereof.

[0085] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0086] The present disclosure will now be described more fully below with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. However, the present disclosure may be presented in many different forms and should not be construed as limited to the embodiments provided herein; on the contrary, these embodiments are provided to make the present disclosure thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art.

[0087] Figure 1 1 is an overview of a data augmentation system (DANR) 100 based on neural rendering according to some embodiments of the present disclosure. The purpose of the system 100 is to augment an object detection dataset with new view images and use the augmented dataset to improve the performance of the object detector. The number of new view images or augmented images is controllable, and the degree of camera position change of the augmented images is also controllable. Figure 1 As shown, the input of the system 100 is a configuration of a labeled dataset 102 and a new camera position 108 for object detection training. The labeled dataset 102 has a plurality of training images 104, each training image 104 having its corresponding annotation 106. The annotation 106 of each training image 104 may include a bounding box surrounding an object in the image and a label of the object. For example, the image may show a pistol, and the annotation of the image may include a bounding box surrounding a pistol in the image, and a label "pistol" or "weapon". The new camera position 108 may be predefined, which may include one or more orientations with different variations relative to the camera position of the image 104. When the dataset 102 and the new camera position 108 are input to the offline dataset enhancement system 110, the neural rendering module 112 renders the image 104 located at a different new camera position 108, thereby obtaining a new image. The bounding box of the image 104 is also rendered together with the image 104, thereby obtaining a rendered bounding box. The rendered bounding box inherits their labels. The new image with the rendered bounding box and the label is formatted by the formatter 114 to obtain an enhanced dataset 116. The enhanced dataset 116 has new images 118 and automatically generated annotations 120. The format of the enhanced dataset 116 is the same as the format of the original training dataset 102. In some embodiments, the system combines the dataset 102 and the enhanced dataset 116 to obtain a new training dataset. Therefore, the new training dataset contains more training data and can be used to effectively train object detection.

[0088] Figure 2 The operation of the new view synthesis model 200 according to some embodiments of the present disclosure is schematically illustrated. In some embodiments, the new view synthesis model corresponds to Figure 1 The neural rendering module 112 in FIG. Figure 2As shown, the model takes as input a red-green-blue (RGB) image 202 (I) and an annotation (B) of a bounding box 204. In some embodiments, the bounding box 204 is represented by a series of two-dimensional (2D) image keypoints (Bi). The RGB image 202 is fed simultaneously to two branches: (1) an hourglass network 206 for depth estimation (Newell et al., Stacked hourglass networks for human pose estimation, ECCV, 2016, the entire contents of which are incorporated herein by reference), and (2) an encoder 208 with Inception-ResNet blocks to extract visual features (Ning et al., Knowledge-guided deep fractal neural network for human pose estimation, the entire contents of which are incorporated herein by reference). The extracted visual features are pixel-aligned, that is, each pixel on the feature map carries a latent space vector corresponding to its specific spatial position. The depth estimation branch assigns a depth value to each pixel and lifts the latent space vector to a point cloud 210. Each point in the point cloud belongs to a three-dimensional (3D) space with a 2.5D coordinate representation (u, v, d) (Iqbal et al., Hand pose estimation via latent 2.5d heatmap regression, ECCV, 2018, the entire contents of which are incorporated herein by reference), where u and v are the 2D coordinates of the point in the point cloud and d is the depth of the point. Each point can unfold its hidden N-dimensional latent space vector after being projected to the target camera view. These feature vectors can be reprojected to an arbitrary camera coordinate system based on a given projection matrix (translation vector T and rotation vector R of the corresponding camera relative to the original image). The camera position of the original image is defined as P. Given a target camera position P′, the present disclosure spreads the feature points according to the method of Wiles (Wiles et al., Synsin: End-to-end view synthesis from a single image, CVPR, 2020, the entire content of which is incorporated herein by reference) to make the rendering process differentiable. After spreading on a palette of size K×K, the points produce an N-dimensional feature map 212 of a predetermined resolution. The feature map 212 is then fed to a decoder network 214 with an inception-residual block and an upsampling layer to render the final new view image 216 (I′). In order to recover the bounding box annotations of the object detection dataset, the present disclosure pre-marks the key points of the bounding box.When they are projected into 3D space with depth information and reprojected back to 2D in the new camera view, the labels always follow the corresponding pixel-aligned points. Therefore, a new bounding box 218 (B′) is automatically defined and labeled. The entire system is end-to-end differentiable. When the original camera position P is used for scattering and decoding, a new image 220 with annotations is obtained. The new image 220 and its bounding box annotations can be compared with the original image 202 and its bounding box annotations to train the model. It should be noted that the annotation of the bounding box includes the location of the bounding box and its label. The location of the bounding box can be defined by its upper left and lower right corners, or by the upper left corner and the height and width of the bounding box, or by the center point, height and width, or by any other suitable format. The label of the bounding box can be the name or type of the object in the bounding box, such as a dog.

[0089] Figure 3 The implementation of the data enhancement based on neural rendering (DANR) system 300 according to some embodiments of the present disclosure is schematically shown. Figure 3 As shown, system 300 includes a computing device 310. Computing device 310 can be a server computing device, a cluster, a cloud computer, a general-purpose computer, or a dedicated computer that provides object detection training and performs object detection. In some embodiments, computing device 310 is a server of an e-commerce platform, which is configured to store training images for object detection, enhance the number of training images, train an object detection model using the enhanced training images, and detect objects in the e-commerce platform using the trained object detection model. In some embodiments, the training images are updated regularly and the object detection model can be retrained regularly.

[0090] like Figure 3As shown, the computing device 310 may include, but is not limited to, a processor 312, a memory 314, and a storage device 316. In some embodiments, the computing device 310 may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripherals. The processor 312 may be a central processing unit (CPU) configured to control the operation of the computing device 310. The processor 312 may run an operating system (OS) or other application programs of the computing device 310. In some embodiments, the computing device 310 may have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 314 may be a volatile memory, such as a random access memory (RAM), for storing data and information during the operation of the computing device 310. In some embodiments, the memory 314 may be a volatile memory array. In some embodiments, the computing device 310 may run on more than one memory 314. Storage device 316 is a non-volatile data storage medium for storing the OS (not shown) and other applications of computing device 310. Examples of storage device 316 may include non-volatile memory, such as flash memory, memory card, USB drive, hard drive, floppy disk, optical drive, solid state drive (SSD), or any other type of data storage device. In some embodiments, storage device 316 may be local storage, remote storage, or cloud storage. In some embodiments, computing device 310 may have multiple storage devices 316, which may be the same storage devices or different types of storage devices, and the application of computing device 310 may be stored in one or more storage devices 316. In some embodiments, computing device 310 is a cloud computer, and processor 312, memory 314, and storage device 316 are shared resources provided on demand through the Internet.

[0091] like Figure 3As shown, the storage device 316 includes an object detection application 318, a training data set 350, a camera position 360, and an enhanced training data set 370. The object detection application 318 includes a data enhancement training module 320, a data enhancement module 322, an enhancement model 324, an object detector training module 338, an object detector 340, and a user interface 342. The object detection application 318 is configured to train the enhancement model 324, obtain an enhanced training data set 370 from the training data set 350 using the trained enhancement model 324, train the object detector 340 using the enhanced training data set 370, and perform object detection on the image using the trained object detector 340. Specifically, the data enhancement training module 320 is configured to instruct the enhancement model 324 to train the model to obtain the trained enhancement model 324, and the data enhancement module 322 is configured to, after receiving a notification that the model of the enhancement model 324 has been trained, instruct the training number of the enhancement model 324 to use the training data set 350 to obtain the enhanced training data set 370.

[0092] The enhanced model 324 is configured to train the enhanced model to obtain a trained enhanced model after receiving an instruction from the data enhancement training module 320, and notify the data enhancement module 322 that the enhanced model 324 has been trained. The enhanced model 324 is also configured to perform data enhancement to obtain an enhanced training data set 370 after receiving an instruction from the data enhancement module 322. In some embodiments, the data enhancement training module 320 and the data enhancement module 322 can also be part of the enhanced model 324.

[0093] like Figure 3As shown, the enhanced model 324 includes a data preparation module 326, a feature encoder 328, a depth estimator 330, a point cloud module 332, a scatter module 334, and an image decoder 336. The data preparation module 326 is configured to prepare data from the training data set 350 and send the prepared data set to the feature encoder 328 and the depth estimator 330. As described above, each data set of the training data set 350 includes an image and an annotation of the image. The image can be an RGB image. In some embodiments, the image can be, but is not limited to, an image with a resolution of 512×512. The annotation includes the position of the border in the image, the border surrounds an object, and the annotation includes a label of the object. The position of the border can be in any appropriate form. In some embodiments, the border is defined by marking each pixel in the border. In some embodiments, the border is defined by a center pixel, a width of the border, and a length of the border. In some embodiments, the border is defined by the four corner points of the border. The label of the border can be the category of the object, or the name of the object. In some embodiments, the object is a pistol, and the label can be "pistol" or "weapon". In some embodiments, the dataset including images and annotations is stored in a JavaScript object notation (JSON) format. In some embodiments, the dataset used to train the enhanced model 324 and the dataset used for enhancement can be different or the same. In some embodiments, the training dataset can be updated regularly, and the enhanced model 324 can train the model after each update. In some embodiments, the dataset used for enhancement can be regularly updated at predefined time intervals, such as daily, weekly, or monthly, and the enhanced model 324 can generate an updated enhanced training dataset 370 after each update.

[0094] The feature encoder 328 is configured to extract visual features from the image in the data set after receiving the prepared data set from the data preparation module 326, and send the visual features to the point cloud module 332. In some embodiments, the visual features are represented by feature vectors, each feature vector corresponding to a pixel in the image. In some embodiments, the feature vector has multiple dimensions, for example, 10 dimensions. In some embodiments, as Figure 4D As shown, the feature encoder 328 includes one or more inception-residual blocks. Each inception-residual block combines the inception design and the ResNet design, that is, the channel-level concatenation (Concat) of the two branches and the pixel-level addition (Add) with the recognition branch. Figure 4A Schematic illustration of an inception-residual block with identity layer to maintain feature resolution. Figure 4BSchematic diagram of an inception-residual block with an upsample layer to increase feature resolution. Figure 4C Schematic diagram of an inception-residual block with reduced feature resolution with an average pooling (AvgPool) layer. Figure 4D Schematically illustrates a feature encoder 328 stacked from an inception-residual block according to some embodiments of the present disclosure, where ID stands for identity. Figure 4A-4E The H and W in the image are the height and width, that is, the resolution of the image. The number after M, or H and W is the number of channels. Figure 4D and Figure 4E As shown in Figure 1, there are three types of inception-residual blocks: ID, UP, and DOWN. Each of the three types of blocks contains an inception design and a residual design. The difference between ID, UP, and Down is whether the block maintains, increases, or decreases the resolution.

[0095] The depth estimator 330 is configured to estimate the depth for each pixel in each image of the dataset after receiving the prepared dataset from the data preparation module 326, and send the depth of the pixel to the point cloud module 332. In some embodiments, the depth of each pixel is represented by a value d, which refers to the distance from the pixel to the camera that captured the image. In some embodiments, the value of the depth d can be normalized by a mathematical transformation, for example, d=1 / log(d), to simplify the training of the neural network. In some embodiments, the depth estimator 330 is an hourglass network, and this UNet-like structure has been shown to be advantageous in exploring global interdependencies across multiple scales. Estimating the depth of a pixel requires understanding global and local features, which is important for perceiving relative depth. In some embodiments, the present disclosure stacks two 4-level hourglass networks with a maximum number of channels of 256 and uses a 1×1 filter in the last convolutional layer to generate a depth map. Figure 4F The architecture of the depth estimator 330 according to some embodiments of the present disclosure is schematically shown. Specifically, the present disclosure simplifies the original hourglass network by replacing the "pooling layer + ResNet block" scheme with a downsampling block, which is composed of Leaky ReLu, convolution (stride 2 or 4, padding 1, kernel size 4) and batch normalization layer (BatchNorm) in this order. The upsampling block is composed of ReLu, 2× or 4× bilinear upsampling, convolution (stride 1, padding 1, kernel size 3) and batch normalization layer in this order. It should be noted that in Figure 4FIn the example, the Up and Down modules increase and decrease the resolution respectively. However, they are not inception-residual blocks. Instead, they are residual blocks with pooling and upsampling layers. Due to the complexity of the hourglass network, it is advantageous to use residual blocks, avoiding the risk of running out of memory during training when using inception-residual blocks. The present disclosure also concatenates the output feature maps of the stacked hourglass subnetworks with the regression depth of the last layer (without batch normalization layer).

[0096] The point cloud module 332 is configured to form a point cloud using the visual features and depth of the pixels when receiving the visual features from the feature encoder 328 and the depth from the depth estimator 330, and provide the point cloud to the scatter module 334. Specifically, for each image, each pixel in the image corresponds to a point in the point cloud. Each point in the point cloud is characterized by its feature vector and depth. In addition, the border is defined in the image, so the features of the border are also defined in the point cloud. For example, each pixel within a border can be marked as within the border, and the points in the point cloud corresponding to the pixels within the border are also marked as part of the border. Accordingly, each point in the point cloud is characterized by its 2.5D position, its feature vector, and a border indicator. The 2.5D position of a point includes its two-dimensional coordinates and depth, where the two-dimensional coordinates of the point are the same as the two-dimensional coordinates of the corresponding pixel in the image. The feature vector has multiple dimensions to refer to different features of a pixel in the image, such as color, edges, and even features of adjacent pixels. As Figure 4D As shown, the feature vectors are learned from the encoder network. Although these features are not easy to interpret, they should retain information about colors, edges, local and global patterns that are perceptible to the human mind. In some embodiments, the length of the feature vector is, for example, 64. The border indicator shows whether a point in the point cloud is within the border, which is consistent with whether the corresponding pixel in the image is within the border.

[0097] When the data augmentation training module 320 instructs the augmentation model 324 to train the augmentation model, the scatter module 334 is configured to use the camera position corresponding to the image in the data set and the point cloud received from the point cloud module 332 to project the point cloud onto a predefined palette to obtain a feature map, and send the feature map to the image decoder 336. In some embodiments, the camera position of the image is predefined. The camera position includes intrinsic and extrinsic parameters. In some embodiments, the intrinsic parameters are provided in the training data annotation during training DANR; and in the test / inference / enhancement process, the intrinsic parameters are used as the default value (original position), that is, f x =f y =1, c x =c y=1. In some embodiments, no external parameters are used because the present disclosure only requires the rotation vector R and the translation vector T from the original orientation. The present disclosure only requires the difference between the original orientation and the target orientation. For example, during enhancement, the present disclosure uses predefined R and T. In some embodiments, the predefined target orientation or new orientation includes two to ten different orientations. In some embodiments, the number of predefined new orientations is four, and the four new orientations share the same translation vector [0 0 0.3], and their respective rotation vectors are [-0.1 -0.15 0], [0.1 -0.15 0], [-0.1 0.15 0], [0.1 0.15 0]. As Figure 5B As shown, the translation vector and the rotation vector together determine the projection matrix.

[0098] The image decoder 336 is configured to decode the feature map to obtain a target image after receiving the feature map, calculate the loss based on the difference between the original image of the training data set 350 and the corresponding target image, and adjust the parameters of the parameter enhancement model 324. Figure 4E Schematically illustrates a decoder structure according to some embodiments of the present disclosure, which is stacked by inception-residual blocks. The decoder structure can be similar to Figure 4D When used with the encoder shown in FIG. 1 , the decoder structure may correspond to the functionality of the interspersed module 334 and the image decoder 336. Figure 4E In , ID is the identity, Up is upsampling, and Down is average pooling downsampling. The resolution of the decoder input features (H′, W′) defaults to Figure 4D The features extracted by the encoder shown in are the same. When the 3D points (2.5D points) are spread to a larger palette, for example, resolution × 2, the feature map is upsampled by a factor of 2, which will increase the resolution of the image rendering. Unlike PixelSS, which feeds the rendered image to an external super-resolution network, this disclosure refers to this super-resolution method as PointSS.

[0099] In some embodiments, the point projection of the scatter module 334 and the image decoder 336 is performed as follows. The present disclosure fixes the intrinsic parameters of all images, assuming that they are taken with a default camera. Specifically, the present disclosure sets the focal length on the x-axis and y-axis to 1 and the skew to 0, while assuming that the principal point is at the origin of the coordinates. Given a 2.5D point representation, the present disclosure maps it to the camera coordinate system (CCS) of the input image:

[0100]

[0101] The CCS is currently considered to be the World Coordinate System (WCS). Given a specific target view, the present disclosure can determine the extrinsic parameters. The conversion from the WCS to the new CCS is direct using R and T:

[0102] P c =R·(P w -T) (2)

[0103] Among them, P c =(X c , Y c , Z c ).

[0104] Then point P c Scatter to the destination camera palette with new pixel locations:

[0105]

[0106] Within each bounding box i, points (at least the four corner points) are labeled with index value i. These labeled keypoints are a subset of S×S points, where S is the resolution of the extracted feature map. When the keypoints from the i-th bounding box are reprojected to the target view after the above process, their new pixel positions are denoted as P′ i ={(u′ i , v′ i )}. The annotation of bounding box i in the target image is calculated as follows:

[0107]

[0108] For each pixel in the bounding box in the original image, there is a corresponding point in the point cloud, which can be named as a bounding box point. For each bounding box point, there is a corresponding projected pixel in the target image, which can be called a bounding box pixel. The bounding box pixels in the target image can then be used to define the projected bounding box in the target image. The projected bounding box can be labeled just like the bounding box in the original image. In this way, the projected bounding box can be easily determined and labeled directly without any object detection process. It should be noted that the projected bounding box and the bounding box in the original image may be slightly different. For example, due to the pixel movement during the projection process, the corner pixels of the bounding box in the original image and the corner pixels in the projected bounding box may be different. In addition, due to the different camera perspectives, some pixels within the bounding box in the image may also be lost in the target image.

[0109] In some embodiments, the loss function used to train the enhanced model 324 is as follows. The present disclosure adopts the discriminator design of Ning (Ning et al., Knowledge-guided deep fractal neural networks for human pose estimation, IEEE transactions on multimedia, 2018, V20 (5), 1246-1259, the entire contents of which are incorporated herein by reference), and uses two multi-layer discriminators, in which the discriminator with lower resolution serves as an intermediate supervision to help the discriminator with higher resolution learn better and faster. The overall loss consists of image-level L1 loss, feature-level content loss, and discriminator loss. In the case where the present disclosure does not enforce supervised learning, the depth map is learned implicitly.

[0110]

[0111] In some embodiments, the training of the enhancement model 324 can be performed using a batch training dataset, where each batch can include, for example, 10-100 images and their annotations. In some embodiments, the training of the enhancement model 324 can be performed a predetermined number of times using the same set of training datasets 350.

[0112] When the enhancement model 324 is trained and the data enhancement module 322 instructs the enhancement model 324 to enhance the training data set, the scatter module 334 is configured to project a point cloud to a predefined color palette using a predefined camera position 360 to obtain a plurality of feature maps, and the image decoder 336 is configured to decode the plurality of feature maps into a plurality of target images, and combine the target images with bounding box annotations and the original images with bounding box annotations to form an enhanced training data set 370. In some embodiments, the camera position 360 includes four different camera positions, and each original training image will produce four target images. Accordingly, the number of data sets in the final enhanced training data set 370 is five times the number of data sets in the training data set 350.

[0113] In some embodiments, the 3D point cloud in the new view is depth sorted using a z-buffer before being scattered onto the palette. For all points in the new view, the point with the closest depth is selected (by popping the z-buffer) to color the point. The present disclosure employs a differentiable renderer to provide gradients with respect to the position of the point cloud. In the differentiable renderer, the K closest points to each pixel are scattered; each of these points affects a range of radius r from the scattered pixel, with the influence proportional to the Euclidean distance to the center of the range. The sorted points are then accumulated using Linear Alpha Over-compositing. In this way, the hard z-buffer becomes differentiable. In some embodiments of DANR, K=128 and r=4. An example of a differentiable renderer is shown in Figure 5A In this example, two points from the z-buffer are spread onto the palette with a radius of influence of 3.

[0114] Object detector training module 338 is configured to train object detector 340 using enhanced training dataset 370 when enhanced training dataset 370 is obtained.

[0115] The object detector 340 is configured to process the image provided by the e-commerce platform to obtain the objects in the image and the annotations of the objects when trained by the object detector training module 338, and report the image and its annotations if the annotations meet certain criteria. In some embodiments, the object detector 340 is configured to use keypoint-based object detection.

[0116] Keypoint estimation is naturally a regression problem. The object is represented by a series of heatmaps, with each channel corresponding to a specific keypoint type. Recently, object detection methods such as CenterNet, CornerNet, and ExtremeNet have begun to utilize keypoint estimation techniques. For example, CenterNet transforms the task of object detection from generating and classifying proposals to predicting object centers (keypoints) and corresponding attributes. For object detection, the attributes are the width, height, and local offset of the object to restore the original resolution pixel position from the downsampled heatmap. Due to the simplicity of CenterNet, this disclosure uses CenterNet as the baseline detection framework for experiments and ablation studies. Because it is anchor-free and does not require non-maximum suppression (NMS) as a post-processing step, ablation studies can be separated from complex design choices that do not care about training data. Anchor-free: During training, there is no need to assign appropriate anchors to objects. Instead, only heatmaps of the entire image are generated as regression targets. No NMS: When inferring heatmaps of object centers, local peaks are sorted according to their responses, and only the top K objects are extracted. Using the center position, corresponding attributes are extracted across their respective channels at the same 2D position. Since different types of key points can appear at the same position, CenterNet is also naturally compatible with multi-label problems.

[0117] In some embodiments, the object detector 340 uses keypoint-based blending for object detection. First, Inoue (Inoue, Data augmentation by paring samples for images classification, arXiv:1801.02929, 2018) showed how mixed samples can be developed into an effective enhancement strategy. Zhang (Zhang et al., Bag of freebies for training object device detection neural networks, arXiv:1902.04103, 2019) described image blending for target detection tasks, but was limited to border blending. In some embodiments, the present disclosure designs a direct image blending strategy for a keypoint-based object detection method, in which a ground truth keypoint is placed on a heat map using a Gaussian kernel.

[0118]

[0119] σ pis an adaptive standard deviation proportional to the object size. During blending, the confidence of the keypoint heatmap is weighted the same as its corresponding image. This blending strategy can be applied to keypoint-based object detection methods such as CenterNet. Compatible with the offline enhancement method of this disclosure, image blending is used as the online enhancement method in all our experiments.

[0120] User interface 342 is configured to provide a user interface or graphical user interface in computing device 310. In some embodiments, a user or administrator of the system can configure parameters for modules in object detection application 318.

[0121] The training data set 350 includes a data set used to train the enhanced model 324 or used by the enhanced model 324 to generate an enhanced training data set 370. The training data set 350 includes images and labels of the images. The label of each image may include a frame surrounding an object in the image and a category of the object. In some embodiments, the training data set 350 may also include a camera position used to shoot the image.

[0122] Camera positions 360 include definitions of camera positions used by augmentation model 324 to generate augmented training data set 370. In some embodiments, a camera position is a relative pose with respect to an original camera position of an image, and each camera position defined by camera positions 360 is used to generate an augmented image. In some embodiments, camera positions 360 are defined so that they are far from each other. The distribution of camera positions at different viewing angles improves the quality of augmented training data set 370. In some embodiments, the number of camera positions 360 is in the range of 2-10. In some embodiments, the number of camera positions 360 is four. Figure 5B Four camera positions 360 are schematically shown. Figure 5B In the example, the camera position 360 is defined by a rotation and a translation relative to the original camera position. In some embodiments, the original camera position can be derived based on intrinsic and extrinsic parameters of the camera of the training images.

[0123] The enhanced training dataset 370 is generated by the enhanced model 324 based on the training dataset 350 and the camera positions 360. In some embodiments, for each training image of the training dataset 350, the enhanced model 324 generates a number of new images corresponding to the number of camera positions 360, and then combines the original training images and the new images with the annotations to form the enhanced training dataset 370.

[0124] It should be noted that the modules in the object detection application 318 can process each image one by one, or can process them in batches, and the images in each batch can be processed in parallel.

[0125] Figure 6The process 600 for training an enhanced model according to some embodiments of the present disclosure is schematically shown. In some embodiments, the training process is performed by a computing device (e.g. Figure 3 The computing device 310 shown in FIG. 3 is specifically executed by the enhanced model 324). It should be noted that, unless otherwise specified in the present disclosure, the steps of the training process can be arranged in different orders, and are therefore not limited to the following. Figure 6 in the order shown.

[0126] like Figure 6 As shown, in step 602, the data enhancement training module 320 calls the training of the enhancement model 324. When the training is initialized, the data preparation module 326 prepares the training data set 350 and sends the prepared training data set to the feature encoder 328 and the depth estimator 330. The prepared training data set 350 includes a plurality of training samples, each of which includes an image and its annotations. The annotations include the definition of the frame surrounding the object in the image, and the identification of the object. In some embodiments, the definition of the frame may include the center, width and length of the frame. In some embodiments, the definition of the frame may be a marker character of the image pixel, wherein the marker character may be inside or outside the frame. The identification of the object may be the category of the object or the name of the object. In some embodiments, the training data set 350 is manually prepared in advance. In some embodiments, the training data set 350 is further automatically expanded by merging new images and their labels screened out in the e-commerce platform. In some embodiments, the training of the enhancement model 324 is performed using a batch training data set. In some embodiments, each batch of training data sets includes 5-50 different training samples. In some embodiments, the number of training samples in each batch is eight. In some embodiments, the training samples are arranged in pairs, and the present disclosure takes a pair of adjacent frames (images) for each back propagation. In each pair, the distance / interval between adjacent images is between 0.5 seconds and 1 second, and the two images belong to the same scene but at slightly different angles. The two images in this pair are named as the first image I and the second image I'. For the first image I and the second image I', the reference data (ground truth) contains their corresponding camera positions.

[0127] In step 604 , for the pair of samples, the present disclosure calculates a rotation vector R and a translation vector T from the first image I to the second image I′ based on their camera positions, and calculates a projection matrix P based on the rotation vector R and the translation vector T.

[0128] At step 606, after receiving the prepared training data set, the feature encoder 328 extracts visual features from the first image I and sends the visual features to the point cloud module 332. In some embodiments, each pixel in the first image I is represented by a visual vector. In some embodiments, each visual vector has, for example, 100 dimensions. In some embodiments, there is one dimension or additional feature for each pixel that indicates whether it is inside or outside the border.

[0129] In step 608 , after receiving the prepared data set, the depth estimator 330 estimates the depth of each pixel in the first image I and sends the depth of the pixel to the point cloud module 332 .

[0130] At step 610, after receiving the visual features and the estimated depth, the point cloud module 332 combines the information of the first image I to form a point cloud of the image, and sends the point cloud to the scatter module 334. The point cloud includes points, each of which corresponds to a pixel in the original image I. Each point includes a two-dimensional coordinate (u, v) and a depth d, has a label indicating whether the point is within the bounding box, and has a visual vector corresponding to the pixel feature, wherein the feature includes color, edge, etc.

[0131] In step 612, after receiving the point cloud, the scatter module 334 uses the projection matrix P to project the points of the point cloud to a predefined palette to obtain a feature map, and sends the feature map to the image decoder 336. Points with features inside or outside the border are also transferred to the feature map. In some embodiments, the present disclosure selects a first image I as an input image, and projects the points of its point cloud using the projection matrix P to obtain a corresponding feature map.

[0132] In step 614, after receiving the feature map corresponding to the first image I, the image decoder 336 decodes the feature map to form a target image I". The target image I" includes a definition of a target border. In some embodiments, each pixel in the target image has a feature inside or outside the border, which is transferred from the feature map.

[0133] In step 616, the image decoder 336 also compares the target image I″ with the second image I′ to calculate the loss, and adjusts the parameters of the enhancement model 324 based on the loss through back propagation. In some embodiments, the present disclosure may also project the first image according to the image's own camera position (using the projection matrix

[000] ) to obtain the target image, calculate the loss by comparing the target image with the first image, and then perform back propagation based on the loss.

[0134] Using the adjusted parameters, the training process may be performed again using another pair of training samples from the training dataset until the parameters converge, or until a predetermined number of training iterations are completed, or until training is performed using a predetermined number of images.

[0135] In some embodiments, for each iteration, the present disclosure uses multiple pairs of images in batches. Using batches in pairs instead of a pair can speed up the training process and average errors. For example, each batch can include eight pairs of images, which can be processed in parallel by, for example, a graphics processing unit (GPU), and the forward propagation in the batch can be averaged; then at the end of the batch processing, the parameters are adjusted and back propagation is performed. After a batch of iterations, the present disclosure then retrieves another batch to perform another iteration of training. In some embodiments, the training data set includes continuous video frames (images), and each pair of images can be selected from the video. The selection of a pair of images can be performed in the following manner: select a frame (image) i from a randomly selected video, and search for a frame in the N frames after frame i, where N is a positive integer, such as 15. The search can be performed in the following manner: select a frame from the 15 frames, calculate the rotation and translation matrix of the selected frame relative to frame i, and when the rotation between the two frames is greater than 5 degrees and the translation is greater than 0.15, determine that the selected frame forms a pair with frame i. The frame selection can be performed frame by frame from the first frame of the 15 frames to the last frame of the 15 frames until a frame that meets the above criteria is found. In some embodiments, the present disclosure can also calculate the rotation and translation matrices of the 15 frames in parallel and randomly select one of the frames that meets the above criteria.

[0136] When the enhanced model 324 is trained, the trained model 324 can be used to enhance the training data set 350 to generate an enhanced training data set 370. In some embodiments, the training data set used to train the enhanced model 324 and the training data set used to generate the enhanced training data set 370 can be different data sets or the same data set. In some embodiments, a portion of the training data set 350 is used to train the enhanced model 324, and the rest of the training data set 350 is used to generate the enhanced training data set 370.

[0137] Figure 7 The process 700 for enhancing a training data set according to some embodiments of the present disclosure is schematically shown. In some embodiments, the enhancement process is performed by a computing device (e.g. Figure 3 The computing device 310 shown in FIG. 3 is specifically executed by the enhanced model 324). It should be noted that, unless otherwise specified in the present disclosure, the steps of the training process can be arranged in different orders, and are therefore not limited to Figure 7 The order shown in .

[0138] like Figure 7 As shown, in process 702, the data enhancement module 322 calls the enhancement model 324 to enhance the training data set. First, the data preparation module 326 prepares the training data set 350 and sends the prepared training data set to the feature encoder 328 and the depth estimator 330. The prepared training data set 350 includes multiple data sets, each of which includes an image and its annotations. The annotations include the definition of the bounding box surrounding the object in the image, and the identification of the object.

[0139] At step 704, upon receiving the prepared training data set, the feature encoder 328 extracts visual features from the images of the data set and sends the visual features to the point cloud module 332. In some embodiments, each pixel in one of the images is represented by a visual vector. In some embodiments, each visual vector has, for example, 100 dimensions. In some embodiments, one of the dimensions or additional features of each pixel corresponds to a bounding box, wherein each pixel is indicated as a pixel inside the bounding box or a pixel outside the bounding box.

[0140] At step 706 , upon receiving the prepared data set, the depth estimator 330 estimates the depth of each pixel in each image and sends the depth of the pixel to the point cloud module 332 .

[0141] At step 708 , after receiving the visual features and estimated depth from one of the images, point cloud module 332 combines the information from each image to form a point cloud for that image and sends the point cloud to scatter module 334 .

[0142] As shown above, processes 704-708 are substantially the same as processes 606-610, except that now the enhancement model 324 has been trained and the prepared dataset may include more data.

[0143] At step 710, upon receiving the point cloud, the spread module 334 retrieves the camera positions 360 (particularly the projection matrix), projects the points of the point cloud to a predefined palette using the projection matrix to obtain feature maps, and sends the feature maps to the image decoder 336. Each feature map corresponds to one of the camera positions 360. If there are four new camera positions, then there will be four generated feature maps for each image.

[0144] At step 712, after receiving the feature map, the image decoder 336 decodes the feature map to form a target image. Each target image corresponds to a feature map. The target image includes a definition of the target border.

[0145] At step 714, the image decoder 336 also combines the target image, the annotation of the target image, and the corresponding original image and its annotation to form an enhanced training dataset 370. For example, if four new camera positions are provided, the model will generate four new target images. Therefore, for each original image, there will be five images with annotations, corresponding to one original image and four new images.

[0146] Figure 8 Schematically illustrates a process 800 for training an object detector using an enhanced training data set according to some embodiments of the present disclosure. In some embodiments, the training process is performed by a computing device (e.g. Figure 3 The computing device 310 shown in FIG. 3 is specifically executed by the object detector training module 338. It should be noted that, unless otherwise specified in the present disclosure, the steps of the training process can be arranged in a different order, and are therefore not limited to Figure 8 Arranged in the order shown in .

[0147] like Figure 8 As shown, when the enhanced training dataset 370 is available, the object detector training module 338 trains the object detector 340 using the enhanced training dataset 370 in process 802. The training may be performed iteratively until the parameters of the object detector 340 converge.

[0148] At step 804, after the object detector 340 is trained, the object detection application 318 retrieves images for detection, and detects objects from the retrieved images using the object detector 340. The detection results may include one or more bounding boxes in the retrieved images, each bounding box enclosing an object in one of the retrieved images, the object having an identification, such as a category or name of the object. In some embodiments, the object detector 340 automatically retrieves images from an e-commerce platform, such as new product images or images from new forum posts.

[0149] In step 806, the object detector 340 also provides a warning message when some categories or names are detected from the retrieved image, and sends the warning message to a specific administrator or user.

[0150] In some embodiments, the object detector 340 has a CenterNet structure. In the object detection process, an RGB image is input, and the output includes multiple heat maps or channels, each heat map corresponding to an object, such as a weapon, a cigarette, or a flag. When the pixel value in a heat map is greater than a threshold (e.g., 0.5), it is determined that the image has an object corresponding to the heat map.

[0151] In some aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. In some embodiments, the computer executable code may be software stored in the storage device 316 as described above. When executed, the computer executable code may perform the above method.

[0152] In some aspects, detection performance is significantly improved using DANR. However, we notice that the AUM drops for smaller objects. We assume that this is due to the smaller rendering resolution (e.g., 256×256) compared to the detector input resolution (e.g., 512×512). To generate higher resolution images, we tried three different settings: (1) Without any further training, the network trained on the smaller image size was applied to a different size at test time. Specifically, the 256 2 The cloud points are scattered onto a 512×512 palette map, which is then rendered as an image with a refinement network. This augmentation method increases the image resolution by supersampling the scattered points and is denoted by 512-pointSS. (2) The image is first augmented at 256×256, and then the pixels are further upsampled at the image level using a pre-trained supersampling network. This method is denoted by 512-pixelSS. (3) A network is retrained from scratch (which receives a 512×512 image), generates a 512×512 feature map, and samples 512 in the cloud. 2 feature points. These points are scattered and rendered on the output image. This method naturally outputs an image with a resolution of 512×512, so it is referred to as 512-native. Fig.9A The performance of DANR using high and low rendering resolutions is compared, where the dataset is the indoor validation set, AP represents the average precision, NA represents the online enhanced data using a keypoint-based image blending strategy, APs represents the AP for small image sizes, and AP M represents the AP of medium image size, AP L Indicates the AP for large image size. Fig.9A As shown in the figure, (1) using images rendered at different resolutions for augmentation can consistently improve detection performance; (2) low-resolution synthetic images may lose some details compared to real images, which can hurt detection performance of very small objects; (3) increasing the image resolution by supersampling can further improve performance; (4) point supersampling may be better than pixel supersampling; (5) training a network with dense features and points achieves the best performance. In the following experiments, 512-native is used as the resolution setting because it achieves the highest AP and does not require any additional supersampling network.

[0153] In some aspects, the performance of DANR is evaluated on datasets with different data scarcity. Experiments are conducted using three data split settings: 5:5, 3:7, and 1:9. The highest scarcity is 1:9. Each image is augmented with four synthetic images. When splitting images into training and validation sets, images and their corresponding augmented images are assigned together. Fig. 9B The performance of DANR is compared using different training: validation data split ratios, where the dataset is an indoor validation set. Fig. 9B As shown in Figure 2, the improvement in performance is closely related to the scarcity of training data. When the annotated data is very limited, the DANR method becomes more advantageous.

[0154] In some embodiments, the backbone of DANR is optimized. The detector is fixed to CenterNet when comparing different backbones. Fig. 9C The performance of DANR using different frameworks (including ResNet-50 and ResNeSt-50) is compared, where the dataset is an indoor validation set. Fig. 9C As shown, the proposed augmentation consistently improves detection performance regardless of the skeleton used by the detector.

[0155] In some embodiments, DANR is evaluated on both primary and secondary detection frameworks. CenterNet is used as an example of a primary method, while Hit-Detector is used to represent a secondary method. The official code of Hit-Detector is used for training and testing. As expected, Fig.9D As shown, DANR does not rely on the one-stage detection framework. It is also beneficial for the two-stage detector.

[0156] In some aspects, experimental results on various datasets show that Fig.9E In. Fig.9E As shown in the figure: (1) Results on the Image Content Review datasets (ICR-Weapon, ICR-Flag, and ICR-Logo) show that DANR is always helpful for real-world data when training data is scarce; (2) Results on the indoor datasets show that augmentation is very beneficial for indoor data, even when training data is not scarce; (3) Results on the COCO dataset show that DANR’s contribution is small when (a) the neural renderer in DANR cannot fully generalize to the semantics of a specific dataset and (b) training data is abundant.

[0157] In summary, some embodiments of the present disclosure provide a new method for data enhancement for object detection based on differentiable neural rendering. Compared with existing methods, neural rendering provides a more promising generative modeling technique for data enhancement because it connects 2D and 3D domains and provides a highly controllable image processing method. During the rendering process, the border definition and labels are retained or projected so that the rendered new image also includes border information. Therefore, the rendered new image and the border information of the new image can be directly used as a training data set for target detection. A large number of experimental results show that the DANR method of the present disclosure significantly improves the detection performance when labeled data is scarce, and is a very promising countermeasure for dealing with long-tail problems and other machine learning paradigms (such as semi-supervised learning). In addition, enhancements based on neural rendering can be effectively applied to other cognitive visual tasks.

[0158] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0159] The embodiments selected and described above are intended to explain the principles of the present disclosure and its practical application, so that other persons skilled in the art can utilize the present disclosure and various embodiments and various modifications suitable for the intended specific use. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure belongs without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is defined by the appended claims rather than by the above description and the exemplary embodiments described therein.

Claims

1. A system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein: The computer executable code, when executed at the processor, is for: Providing a two-dimensional (2D) image and a projector position, wherein the 2D image includes a frame, the frame surrounds an object, and the frame is labeled according to the object; Extracting feature vectors of pixels in the 2D image, each pixel having a corresponding feature vector; estimating a depth of the pixel from the 2D image; generating a point cloud using the feature vector and the depth of the pixel; and Projecting the point cloud using the projector to obtain a projection image, Among them, each pixel within the border has a border label and is called a border pixel, each point in the point cloud corresponding to the border pixel has the border label and is called a border point, each image pixel in the projected image corresponding to the border point has the border label and is called a projection border pixel, and the projection border pixels are used to define the projection border.

2. The system of claim 1, wherein the computer executable code is for extracting feature points using an encoder having an Inception-ResNet block.

3. The system of claim 1, wherein the computer executable code is for estimating the depth using an hourglass network.

4. The system of claim 1, wherein when the projection position is the original position of a camera that captures the 2D image, the computer executable code is configured to project the point cloud using the following formula: in, (u, v) is the coordinate of the target point in the point cloud, d is the depth of the target point in the point cloud, c x and c y is the offset of the center of the camera, f x and f y is the focal length of the camera on the x-axis and y-axis, and (X c , Y c , Z c ) are the coordinates of the target point in the camera coordinate system.

5. The system of claim 4, wherein the computer executable code is for comparing the projected image with the 2D image to learn parameters for extracting feature key points, estimating the depth, generating the point cloud, and projecting the point cloud.

6. The system according to claim 4, wherein when the projection position is a new position with a rotation R and a translation T relative to the original position, the computer executable code is used to project the point cloud using the following formula: P c =R·(P w -T) formula (2), and in, P w Equivalent to the above formula (1), P c is the coordinate of the target point in the new camera coordinate system corresponding to the new camera position, P c By (X′ c , Y′ c , Z′ c ) denotes, (u′, v′) are the coordinates of the pixel in the projection image corresponding to the target point.

7. The system of claim 6, wherein the projection bounding box is defined by the following formula: Where P′ i ={(u′ i , v′ i )} are the coordinates of the pixels in the projection frame.

8. The system of claim 1, wherein the computer executable code is further configured to: combining the 2D image, the frame, the projected image, and the projected frame to form a training data set; Using the training data set to train a neural network for object detection; Using the neural network to detect objects in images retrieved from a commercial platform; When the detected object of one of the retrieved images belongs to a predetermined object type, a warning message is sent for the one of the retrieved images. 9 . The system according to claim 1 , wherein the projection camera positions include four different camera positions different from the original camera position for capturing the 2D image.

10. The system according to claim 1, wherein the number of the projection positions is four, and the four projection positions are evenly distributed in the three-dimensional space relative to the original position.

11. A method comprising: Providing a two-dimensional (2D) image and a projector position, wherein the 2D image includes a frame, the frame surrounds an object, and the frame is labeled according to the object; Extracting feature vectors of pixels in the 2D image, each pixel comprising a corresponding feature vector; estimating a depth of the pixel from the 2D image; generating a point cloud using the feature vector and the depth of the pixel; and Projecting the point cloud using the projector to obtain a projection image, Among them, each pixel within the border has a border label and is called a border pixel, each point in the point cloud corresponding to the border pixel has the border label and is called a border point, each image pixel in the projected image corresponding to the border point has the border label and is called a projection border pixel, and the projection border pixels are used to define the projection border.

12. The method of claim 11, wherein the feature vector is extracted using an encoder having an Inception-ResNet block. The method of claim 11 , wherein the depth is estimated using an hourglass network.

14. The method according to claim 11, wherein when the projection position is the original position of the camera that captures the 2D image, the point cloud is projected using the following formula: in, (u, v) is the coordinate of the target point in the point cloud, d is the depth of the target point in the point cloud, c x and c y is the offset of the center of the camera, f x and f y is the focal length of the camera on the x-axis and y-axis, and (X c , Y c , Z c ) are the coordinates of the target point in the camera coordinate system.

15. The method of claim 14, further comprising comparing the projected image with the 2D image to learn parameters for extracting feature key points, estimating the depth, generating the point cloud, and projecting the point cloud.

16. The method according to claim 14, wherein when the projection position is a new position with a rotation R and a translation T relative to the original position, the point cloud is projected using the following formula: P c =R·(P w -T) formula (2), and in, P w Equivalent to the above formula (1), P c is the coordinate of the target point in the new camera coordinate system corresponding to the new camera position, P c By (X′ c , Y′ c , Z′ c ) denotes, (u′, v′) are the coordinates of the pixel in the projection image corresponding to the target point.

17. The method of claim 16, wherein the projection bounding box is defined by the following formula: Where P′ i ={(u′ i , v′ i )} are the coordinates of the pixels in the projection frame.

18. The method according to claim 11, further comprising: combining the 2D image, the frame, the projected image, and the projected frame to form a training data set; Using the training data set to train a neural network for object detection; Using the neural network to detect objects in images retrieved from a commercial platform; When the detected object of one of the retrieved images belongs to a predetermined object type, a warning message is sent for the one of the retrieved images.

19. The method according to claim 11, wherein the projection camera positions include four different camera positions different from the original camera position for shooting the 2D image.

20. A non-transitory computer readable medium storing computer executable code, wherein: The computer executable code, when executed at a processor of a computing device, is used to: Providing a two-dimensional (2D) image and a projector position, wherein the 2D image includes a frame, the frame surrounds an object, and the frame is labeled according to the object; Extracting feature vectors of pixels in the 2D image, each pixel having a corresponding feature vector; estimating a depth of the pixel from the 2D image; generating a point cloud using the feature vector and the depth of the pixel; and Projecting the point cloud using the projector to obtain a projection image, Among them, each pixel within the border has a border label and is called a border pixel, each point in the point cloud corresponding to the border pixel has the border label and is called a border point, each image pixel in the projected image corresponding to the border point has the border label and is called a projection border pixel, and the projection border pixels are used to define the projection border.

Images