Visual-based perception systems

Abstract

A vision-based perception system for monitoring a physical environment, comprising a LIDAR device configured to acquire a time-series point cloud dataset representing the environment, a camera device configured to capture a time-series image of the environment, and a processing unit including a first neural network and a second neural network different from the first neural network. The processing unit is configured to recognize objects present in the time-series image by the first neural network, determine unrecognized objects present in the time-series image, determine a bounding box of the determined unrecognized objects, obtain a neural network representation of the unrecognized objects based on the determined bounding box by the second neural network, and train the first neural network for recognition of the unrecognized objects based on the obtained neural network representation.

Description

[[TECHNICAL FIELD]]

[0001] The present disclosure relates to a vision-based perception system configured for object recognition by a neural network, wherein the vision-based perception system comprises a LIDAR device and a camera device. [[BACKGROUND ART]]

[0002] A vision-based perception system comprising one or more light detection and ranging (LIDAR) devices configured to acquire a time-series 3D point cloud dataset of a detected object, one or more camera devices configured to capture time-series 2D images of the object, and a processing unit configured to recognize the object is used in various applications. For example, vehicles such as automobiles, automated guided vehicles (AGVs), and autonomous mobile robots can be equipped with such a vision-based perception system to facilitate navigation, positioning, and obstacle avoidance. In the context of automobiles, the vision-based perception system can be included in an advanced driving assistance system (ADAS).

[0003] In recent years, deep learning techniques have been used for automated object recognition. Deep learning visual-based perception systems used for object recognition are trained on vast amounts of training data and have proven to reliably recognize objects belonging to classes of objects that contain a large number of objects to be recognized. However, deep learning visual-based perception systems in this art tend to fail to recognize objects that exist in only a few examples in the training dataset (rare objects) and cannot recognize unknown (untrained) objects. Therefore, the training dataset previously used for training is augmented with training samples that represent unrecognized objects. Data augmentation has traditionally been achieved by synthetically creating computer graphics assets that are added to the training set for further training of the visual-based perception system (U.S. Patent Application Publication 2020 / 0051291, U.S. Patent No. 9767565). However, computer graphics assets have traditionally had to be created by experts in a time-consuming and cumbersome manner. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent Application Publication No. 2020 / 0051291 [Patent Document 2] U.S. Patent No. 9767565 [Overview of the project]

[0005] With the foregoing in mind, the underlying objective of this application is to provide a visual-based perception system that can reliably recognize rare objects and learn to recognize unknown objects at low cost. In this application, by definition, “object recognition” is expressly declared to encompass object recognition and / or object identification.

[0006] The aforementioned and other objectives are achieved by the subject matter of the independent claims. Further embodiments will become apparent from the dependent claims, description, and drawings. [Means for solving the problem]

[0007] According to a first embodiment, a visual-based perception system (e.g., deep learning) for monitoring the physical environment of a visual-based perception system is provided, comprising: a light-detecting and ranging LiDAR device configured to acquire a time-series point cloud dataset representing the environment; a camera device configured to acquire a time-series image of the environment; and a processing unit including a first neural network and a second neural network distinct from the first neural network. The processing unit is configured to recognize objects (e.g., multiple objects) present in the time-series image by the first neural network and to determine (detect) unrecognized objects (e.g., unknown / untrained types of vehicles) present in the time-series image. Furthermore, the processing unit is configured to determine a bounding box for the determined unrecognized objects based on at least one of the point cloud datasets, to acquire a neural network representation of the unrecognized objects based on the time-series image and the determined bounding box by the second neural network, and to train the first neural network for recognition of the unrecognized objects based on the acquired neural network representation of the unrecognized objects.

[0008] The processing unit can operate fully automatically or semi-automatically. Neural network representations of unrecognized objects can be obtained without interaction by human operators. The use of a second neural network to obtain a neural network representation of an unrecognized object and to use that neural network representation of the unrecognized object allows for the rapid acquisition of multiple augmented training datasets without requiring the time-consuming and costly manual creation of synthetic assets by human experts.

[0009] Needless to say, in this specification, the term “neural network” refers to an artificial neural network. The first neural network may be a deep neural network, which may include a fully connected feedforward neural network (multilayer perceptron, MLP) or a recurrent or convolutional neural network and / or transformer. The second neural network is used to obtain a neural network representation of an unrecognized object, which may be an MLP, such as a deep MLP, or may include one.

[0010] According to one embodiment, the processing unit of the vision-based perception system is configured to cluster each point in the point cloud dataset to obtain each point cluster in the point cloud dataset, and to determine unrecognized objects by determining, with respect to at least a portion of the point cloud dataset, that one of the point clusters does not correspond to any object present in the time-series image recognized by the first neural network. Thus, unrecognized objects can be reliably and quickly determined based on data provided by the LIDAR device and the camera device.

[0011] According to another embodiment, the processing unit of a visual-based perception system is configured to determine unrecognized objects by performing ground segmentation based on a point cloud dataset to determine the ground, and by determining that the unrecognized object is located on the determined ground. In particular, the processing unit may be configured to determine unrecognized objects only by both determining that, with respect to at least a portion of the point cloud dataset, one of the point clusters does not correspond to any object recognized by the first neural network, and determining that the unrecognized object is located on the determined ground. "Ground segmentation" means the automatic segmentation of the ground on which an object is located from the rest of the image and / or point cloud. Ground segmentation can be performed based on the image, or based on both the image and the LIDAR point cloud. An example of ground segmentation is road segmentation in the context of automotive applications. The adoption of ground segmentation facilitates the reliable determination of unrecognized / unrecognizable objects in images captured by the camera device of a visual-based perception system.

[0012] According to another embodiment of the vision-based perception system of the first aspect, the processing unit is further configured to determine, with respect to each of a plurality of pre-stored images captured by a camera device and a corresponding pre-stored LIDAR point cloud dataset captured by a LIDAR device, at least one other object having a corresponding bounding box determined by the processing unit, based on at least one of the corresponding pre-stored LIDAR point cloud datasets having the same dimensions within a predetermined threshold as the bounding box determined by the processing unit with respect to the determined unrecognized object, and to replace the at least one other object determined in each of the plurality of pre-stored images with a neural network representation of the unrecognized object in order to obtain a plurality of training images for training a first neural network for the recognition of unrecognized objects.

[0013] Pre-stored images may be captured by the same camera device of the first embodiment of the visual-based perception system, and pre-stored LIDAR point cloud datasets may be captured by the LIDAR device of the first embodiment of the visual-based perception system. According to this embodiment, objects present in the pre-stored images are replaced with neural network representations of unrecognized objects in order for the second neural network to create a large number of new training images (augmented training data) representing new training scenes that can be appropriately used to train the first neural network for object recognition.

[0014] To facilitate the training procedure of a first neural network that yields reliable recognition results, lighting and shading conditions may be appropriately taken into account when creating training images. Accordingly, according to another embodiment, the processing unit of the visual perception system is configured to determine, for each of a plurality of pre-stored images captured by a camera device, a first light direction (direction of incident light) for at least one other object, a second light direction (direction of incident light) for the determined unrecognized object, and to replace the determined at least one other object in each of the plurality of pre-stored images with a neural network representation of the unrecognized object when the first light direction and the second light direction are shifted from each other by less than a predetermined illumination (deviation) angle (for example, only in this case). According to this embodiment, an image having at least one other object illuminated from a light direction too different from the light direction for the determined unrecognized object is not considered sufficiently suitable to serve as the basis for training images.

[0015] According to another embodiment, a second neural network of a vision-based perception system includes a first multilayer perceptron MLP trained on neural radiance field technology and a second MLP trained on neural radiance field technology, distinct from the first MLP, wherein the first MLP is configured to acquire a neural network representation of an unrecognized object and the second MLP is configured to acquire a neural network representation of the background of an unrecognized object.

[0016] Neural Radiance Field (NERF) technology was presented at "Computer Vision-ECCV2020" (16 th Proposed by B. Mildenhall et al. in a paper titled "Nerf: Representing scenes as neural radiance fields for view synthesis" at the European Conference, Glasgow, UK, August 23-28, 2020 (Springer, Cham, 2020), NERF enables the efficient acquisition of accurate neural network representations of the environment, particularly unrecognized objects, based on color values ​​and spatially dependent volume density values ​​(see also the detailed explanation below).

[0017] Using a first MLP and a second MLP trained with NERF, unrecognized objects can be automatically separated from the background without human interaction. In one embodiment, some ray splitting technique is used to achieve this separation. In this embodiment, the first MLP is configured to acquire a neural network representation of the unrecognized object based on the portion of the camera ray that crosses the region corresponding to the bounding box, and the second MLP is configured to acquire a neural network representation of the background of the unrecognized object based on the portion of the camera ray that does not cross the region corresponding to the bounding box. Based on bounding boxes acquired from LIDAR point cloud data, the neural network representation of the unrecognized object and its background can be acquired automatically and relatively easily based on individually designed MLPs.

[0018] Training images can be obtained by rendering neural network representations of unrecognized objects onto pre-stored images, for example, by replacing one or more recognized common objects present in the pre-stored images with unrecognized objects. For example, a neural network representation of a scene containing one or more recognized objects can be obtained by a neural network trained on a first NERF and a neural network trained on a second NERF, and at least one other determined object can be replaced with a neural network representation of a rare object. According to one embodiment, the first MLP is configured to obtain a neural network representation of unrecognized objects based on the camera pose (of the camera used to capture images containing unrecognized objects), and the processing unit is configured to replace at least one other determined object with a neural network representation of unrecognized objects by rendering the neural network representation of unrecognized objects onto each of a plurality of pre-stored images based on a rendering pose that deviates from the camera pose by less than a predetermined threshold (for example, only by the rendering pose), and As a result, the position and direction in which rendering is performed can be controlled to ensure that artifacts caused by the position / orientation of unrecognized objects that do not exist in the time-series image of the environment including unrecognized objects can be avoided.

[0019] A visual-based perception system according to the first embodiment and any of its embodiments can be appropriately used in a vehicle. Accordingly, according to another embodiment, the visual-based perception system is configured to be installed in a vehicle, particularly an automobile, an autonomous mobile robot, or an automated guided vehicle (AGV), where a time-series point cloud dataset and a time-series image represent the driving scene of the vehicle. In automotive applications, the visual-based perception system according to the first embodiment and any of its embodiments can be included in an advanced driver-assistance system (ADAS).

[0020] According to a second aspect, a vehicle is provided that includes a vision-based perception system according to the first aspect or any embodiment thereof. The vehicle may be an automobile, an autonomous mobile robot, or an automated guided vehicle (AGV).

[0021] According to a third aspect, a method is provided for training a first neural network of a visual-based perception system for object recognition. This method includes the steps of: acquiring a time-series point cloud dataset representing the environment of the visual-based perception system by a light-detecting and ranging LIDAR device of the visual-based perception system; capturing a time-series image of the environment by a camera device of the visual-based perception system; determining (detecting) unrecognized objects in the time-series image by a processing unit of the visual-based perception system; determining a bounding box for the determined unrecognized object based on at least one of the point cloud datasets by the processing unit; acquiring a neural network representation of the unrecognized object based on the time-series image and the determined bounding box by a second neural network of the visual-based perception system; and training a first neural network for recognition of the unrecognized object based on the acquired neural network representation of the unrecognized object.

[0022] Unrecognized objects may include, for example, an unknown (untrained) brand or type of vehicle, or an unknown object on a street or road.

[0023] According to one embodiment, the method of the third embodiment further includes the step of a processing unit clustering each point of a point cloud dataset in order to obtain each point cluster of the point cloud dataset, wherein unrecognized objects are determined by determining, with respect to at least a portion of the point cloud dataset, that one of the point clusters does not correspond to any object recognized by the first neural network.

[0024] According to another embodiment, the method further comprises the step of, performed by a processing unit, executing ground segmentation (e.g., road segmentation) based on the point cloud data set to determine the ground, wherein an unrecognized object is determined by determining that the unrecognized object is located on the determined ground.

[0025] According to another embodiment, for each of a plurality of pre-stored images captured by a camera device (e.g., a camera device of a vision-based perception system) and corresponding pre-stored LIDAR point cloud data sets captured by a LIDAR device (e.g., a LIDAR device of a vision-based perception system), the method further comprises: determining, by a processing unit, at least one other object having a corresponding bounding box determined by the processing unit based on at least one of the corresponding pre-stored LIDAR point cloud data sets, the bounding box having the same dimensions within a predetermined threshold as the bounding box determined by the processing unit for the determined unrecognized object; and replacing at least one other object determined in each of the plurality of pre-stored images with a neural network representation of the unrecognized object to obtain a plurality of training images for training a first neural network for recognition of the unrecognized object.

[0026] According to another embodiment, the method further comprises the step of, performed by a processing unit, for each of a plurality of pre-stored images captured by a camera device, determining a first light direction with respect to the at least one other object and determining a second light direction with respect to the determined unrecognized object, wherein the determined at least one other object is replaced with the neural network representation of the unrecognized object in each of the plurality of pre-stored images when the first light direction and the second light direction deviate from each other by less than a predetermined illumination (deviation) angle.

[0027] According to one embodiment, the second neural network includes a first multilayer perceptron MLP trained on neural radiance field technology and a second MLP trained on neural radiance field technology different from the first MLP, wherein a neural network representation of an unrecognized object is obtained by the first MLP, and the method further includes the step of obtaining a neural network representation of the background of the unrecognized object by the second MLP.

[0028] According to another embodiment, the neural network representation of an unrecognized object is obtained based on the portion of the camera ray that crosses the region corresponding to the bounding box, while the neural network representation of the background of the unrecognized object is obtained based on the portion of the camera ray that does not cross the region corresponding to the bounding box.

[0029] According to another embodiment, the neural network representation of an unrecognized object is obtained based on the camera pose (of the camera), and at least one other object determined is replaced by the neural network representation of the unrecognized object by rendering the neural network representation of the unrecognized object onto each of a plurality of pre-stored images based on a rendering pose that deviates from the camera pose by less than a predetermined threshold.

[0030] According to a fourth aspect, a method for recognizing objects present in an image is provided, comprising the step of training a first neural network of a visual-based perception system for object recognition, as described in a third aspect and its embodiments.

[0031] A third aspect and its embodiments, a method for training a first neural network of a visual-based perception system for object recognition, and a fourth aspect, a method for recognizing objects present in an image, offer the same or similar advantages as those described above with reference to the visual-based perception system of the first aspect and its embodiments. The visual-based perception system of the first aspect and its embodiments may be configured to perform the methods of the third aspect and its embodiments and the methods of the fourth aspect.

[0032] According to the fifth aspect, a computer program product is provided which, when executed on a computer, includes computer-readable instructions for performing or controlling steps of the third or fourth aspect and the methods according to embodiments thereof.

[0033] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims.

[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying figures and drawings. [Brief explanation of the drawing]

[0035] [Figure 1] This document presents a visual-based perception system according to one embodiment. [Figure 2] This is a flowchart showing a method for training a first neural network of a visual-based perception system according to one embodiment for object recognition. [Figure 3] This demonstrates data augmentation of training data used by neural networks in vision-based perception systems for object recognition. [Figure 4] This demonstrates the ray splitting technique used to create NERF-based composite assets. [Figure 5] This demonstrates the creation of composite assets based on NERF. [Figure 6]This shows a rendering posture suitable for creating composite assets based on NERF. [Figure 7] This demonstrates the replacement of other objects present in pre-stored images with rare objects for the creation of NERF-based composite assets. [Modes for carrying out the invention]

[0036] This specification provides techniques for data augmentation of training datasets used in deep learning vision-based perception systems configured for object recognition. An embodiment of a vision-based perception system 100 for monitoring the physical environment of the vision-based perception system 100 in which such techniques are implemented is shown in Figure 1. The vision-based perception system 100 comprises a LIDAR device 110, for example, multiple LIDAR devices, and a camera device 120, for example, multiple camera devices. Furthermore, the vision-based perception system 100 comprises a processing unit 130 including a first neural network 132 and a second neural network 138. The processing unit 130 is configured to receive input data from both the LIDAR device 110 and the camera device 120.

[0037] The first neural network 132 may include a recurrent or convolutional neural network and / or a transformer, and is trained for object recognition. The first neural network 132 is configured to receive input data based on images captured by the camera device 110. The input data may include patches of images. The input data may include a tensor having shape (number of images) × (image width) × (image height) × (image depth). With respect to the first layer of the neural network that processes the input data, the number of input channels may be greater than or equal to the number of channels in the data representation, for example, three channels for the RGB or YUV representation of the image. By passing through the neural network layers, the image is abstracted into a feature map having shape (number of images) × (feature map width) × (feature map height) × (feature map channels), which can then be further processed. The first neural network 132 may include a neural network that acts as a classifier. The first neural network 132 may also be configured to receive point cloud data acquired by the LIDAR device 110.

[0038] The second neural network 138 is configured to receive input data based on images captured by the camera device 120. The second neural network 138 may include one or more MLPs and may be trained to generate an implicit neural network representation of the environment, including information about the camera device's pose. The second neural network 138 may include one or more NERF-trained neural networks (see description below).

[0039] The LIDAR device 110 is configured to acquire a time-series point cloud dataset representing the system environment, and the camera device 120 is configured to capture time-series images of the environment. The processing unit 130 is configured to recognize objects (e.g., multiple objects) present in the time-series images using a first neural network and to determine (detect) unrecognized objects present in the time-series images. Unrecognized objects may be objects that the first neural network was not trained to recognize, or objects that were present only in a relatively sparse subset of the training data.

[0040] Furthermore, the processing unit 130 is configured to determine the bounding box of the determined unrecognized object based on at least one of the point cloud datasets, and to obtain a neural network representation of the unrecognized object based on the time-series images and the determined bounding box using a second neural network.

[0041] Furthermore, the processing unit 130 of the visual-based perception system 100 is configured to train a first neural network for recognizing unrecognized objects based on acquired neural network representations of unrecognized objects. A second neural network 138 may be used to create a plurality of new training images, each containing a neural network representation of an unrecognized object, and the first neural network 132 may be trained on the plurality of new training images thus created (see also the description below). The adoption of the second neural network 138 allows for the generation of an augmented training dataset suitable for training the first neural network 132 to recognize unrecognized objects without any interaction by human experts, or with at least less interaction than required in the art.

[0042] A method 200 for training a first neural network of a visual-based perception system for object recognition according to one embodiment is illustrated by the flowchart shown in Figure 2. This method can be implemented in the visual-based perception system 100 shown in Figure 1, which may be configured to perform one or more steps included by the method 200 shown by the flowchart shown in Figure 2.

[0043] Method 200 includes the steps of: acquiring a time-series point cloud dataset representing the environment of the visual-based perception system by a LIDAR device (one or more LIDAR devices) of the visual-based perception system; and acquiring a time-series image of the environment by a camera device (one or more camera devices) of the visual-based perception system. Furthermore, Method 200 includes the steps of: determining (detecting) unrecognized objects in the time-series image by a processing unit of the visual-based perception system by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a processing unit by a neural network of the visual-based perception system by a neural network representing the unrecognized objects based on the time-series image and the determined bounding box.

[0044] Furthermore, method 200 includes step S260 of training a first neural network for recognizing unrecognized objects based on the neural network representation of the unrecognized object acquired by the second neural network. In the training process S260, multiple new training images can be created and used, each containing the neural network representation of an unrecognized object.

[0045] Figure 3 illustrates the procedure for data augmentation 300 of training data used by a neural network of a visual-based perception system configured for object recognition. Data augmentation can be achieved in a fully automated manner or in a semi-automated manner that requires some human interaction but less human interaction than is required for synthetic asset creation in the art. In the illustrated example, the visual-based perception system is installed in a car. While the car is in motion, the driving scene is captured by the LIDAR and camera devices of the visual-based perception system installed in the car. The captured driving scene is stored in a driving scene / sequence database in the form of a time-series point cloud dataset acquired by the LIDAR device and a time-series image acquired by the camera device (310). In the rare object retrieval procedure 320, rare objects that cannot be recognized (e.g., rare vehicles) are automatically determined (detected), and a 3D bounding box corresponding to / containing this object is automatically extracted from the Lidar point cloud. The determination of the 3D bounding box is based on clustering of points in the point cloud, which can be obtained by computing a convex hull with respect to the acquired 3D point clusters. The extracted 3D bounding box is defined by its center position (x, y, z), orientation angles (roll, pitch, yaw), and dimensions (width, length, height).

[0046] Determining an unrecognized object may involve determining that a particular cluster of points in a time-series point cloud dataset cannot be associated with an object present in the image that is recognized by a neural network included in a vision-based perception system. Furthermore, determining an unrecognized object may involve determining that such a particular cluster of points is located on a road determined by road classification performed on the time-series point cloud dataset, and / or determining that the unrecognized object is located on a road determined by road classification.

[0047] In the neural (synthetic) asset creation procedure 330, neural assets are created using NERF ("Computer Vision-ECCV2020" (16 th It is created by a neural network trained by B. Mildenhall et al. (see the paper titled "Nerf: Representing scenes as neural radiance fields for view synthesis" presented at the European Conference, Glasgow, UK, August 23-28, 2020, Springer, Cham, 2020).

[0048] Data based on time-series images acquired by a camera device representing a driving scene is input to a neural network trained with NERF. The NERF-trained neural network learns an implicit neural network representation of rare objects. The input data represents the coordinates (x,y,z) and the line of sight direction (θ,φ) corresponding to a set of sampled 3D points, and the NERF-trained neural network outputs a line of sight-dependent color values ​​(e.g., RGB) and a volume density value σ, which can be interpreted as the differential probability of the camera ray ending in infinitesimal particles at position (x,y,z) (see the paper by B. Mildenhall et al. mentioned above). Thus, the MLP has F with optimized weights Θ acquired during the training phase. Θ This achieves the transformation (x,y,z,θ,φ) → (R,G,B,σ). The output color values ​​and volume density values ​​are used for volume rendering to obtain a neural network representation of the driving scene captured by the camera device.

[0049] The extracted bounding box is used to "separate" the rendering of the rare object from the rendering of the rare object's background. The rendered "separated" rare object is a neural network representation of the rare object. By "separating" the rare object based on the extracted bounding box, manual segmentation of the rare object in the image can be avoided, thereby significantly reducing time, cost, and human effort compared to the art.

[0050] According to one embodiment, a NERF-trained neural network includes a first NERF-trained neural network for obtaining a neural network representation of a rare object and a second NERF-trained neural network for obtaining a neural network representation of the background of the rare object. Training these two different NERF-trained neural networks is based on the camera ray division technique shown in Figure 4. Similar to ray marching taught by B. Mildenhall et al., a set of camera rays for a set of camera poses is generated for each (training) image of a time-series of images acquired by a camera device representing a driving scene. As shown in Figure 4, the first portion of the camera rays that crosses the determined rare object (bounding box) is distinguished from the second portion of the camera rays that do not cross the determined rare object (bounding box) but cross its background. The first portion can be used to render the rare object, and the second portion can be used to render its background.

[0051] The neural network representations of rare objects are stored in the neural asset database (340). The procedure described above can be performed on a wide variety of rare objects that cannot be recognized based on the available training data used to train the neural network used for object recognition (different from the neural network trained with NERF 330). Thus, a wide variety of neural network representations of various rare objects can be stored in the neural asset database (340).

[0052] In configuration step 350, training images are acquired based on neural assets (neural network representations of rare objects) and previously stored images captured by one or more camera devices. A previously stored image is selected, and the neural network representation of the rare object is rendered onto that image, resulting in a new training image containing the rare object. The new training neural network representation can be used to train a neural network used for object recognition. A vast number of pre-stored images can be selected, and the rare object can be rendered onto each of the vast number of selected pre-stored images (backgrounds) to acquire a vast number of new training images (360) stored in the augmented data database, resulting in a new and rich training dataset for training a neural network used for object recognition.

[0053] The selection of previously stored images from a vast number of pre-stored driving scene images in configuration step 350 may include: a) determining whether another (e.g., a recognized or recognizable common) object with a corresponding bounding box having the same dimensions as the rare object within a predetermined threshold is present in the image; and b) determining whether the direction of light for the other object is similar to that for the rare object within a predetermined limit (see Figure 5). When these conditions are met, i.e., in scenes with similar bounding boxes and light directions, the other object can be appropriately replaced with a neural asset created for the unrecognized rare object (a minivan in the example shown in Figure 5).

[0054] For example, a neural network representation of a scene containing other objects can be obtained by a neural network trained on a first NERF and a neural network trained on a second NERF, and the other objects can be replaced with neural network representations of rare objects. Taking light direction into account allows for the use of coherent / realistic lighting and shading in the process of creating training images for the augmented dataset by rendering the neural network representation against the background of a pre-stored image, thereby improving / facilitating the training procedure based on the augmented data.

[0055] Furthermore, it must be determined from which position and in which (line of sight) the composite image containing the rare object should be rendered. Creating a composite image that depicts the rare object at a position and orientation very different from that used to obtain the neural network representation of the rare object can result in artifacts that can significantly affect the training procedure based on the augmented data. Therefore, according to one embodiment, an appropriate rendering pose for rendering the desired composite image should be determined by comparing the rendering pose with the pose used to obtain the neural network representation of the rare object (see Figures 5 and 6).

[0056] Figure 6 illustrates an example for conditioning the selection of possible rendering poses. Unrecognized rare objects are detected by a camera device from multiple camera poses while a vehicle equipped with a visual-based perception system is in motion. In a composite image suitable for training the recognition of rare objects, the rare objects should be rendered in an acceptable rendering pose. An acceptable rendering pose is determined based on the percentage of effective rendering rays characterized by an angular deviation to the corresponding camera rays used to obtain a neural representation of the rare object, which does not exceed a predetermined angular deviation threshold. For example, a predetermined angular deviation threshold of 20° to 30° may be considered suitable for effective rendering rays, and rendering poses for rendering rare objects in a new composite image representing augmented data for training purposes may only be acceptable if characterized by a percentage of at least 60% effective rendering rays (see Figure 6). By selecting acceptable poses and appropriate lighting conditions for the rendering process, realistic positioning, line of sight, and illumination of the rendered rare objects can be achieved.

[0057] When a pre-stored image containing other objects, such as a recognizable common object, is considered suitable for generating a new composite image containing a neural network representation of a rare object, the other objects can be replaced with the neural network representation of the rare object. In certain examples, the pre-stored image may be the same as the captured image containing an unrecognized rare object, as shown in Figure 7. A stored actual image captured by one or more camera devices can be converted into a neural network representation of the captured scene.

[0058] The first NERF-trained network neural network learns / acquires an implicit neural network representation of rare objects and selects recognized common objects according to similar lighting conditions also present in the scene for replacement with the neural network representation of rare objects, to acquire a new synthetic training image containing rare objects using different rendering poses at different locations.

[0059] All embodiments described above are not intended to be limiting and serve as examples illustrating the features and advantages of the present invention. It should be understood that some or all of the features described above may be combined in different ways. [Explanation of symbols]

[0060] 100 Visual-Based Perception Systems 110 Lidar Devices 120 Camera Devices 130 processing units 132 The First Neural Network 138 The Second Neural Network

Claims

1. A visual-based perception system (100) for monitoring the physical environment of the visual-based perception system (100), A light detection and ranging LIDAR device (110) configured to acquire a time-series point cloud dataset representing the environment, A camera device (120) configured to capture time-series images of the aforementioned environment, A processing unit (130) comprising a first neural network (132) and a second neural network (138) different from the first neural network, The first neural network (132) recognizes objects present in the time-series images, Determine the unrecognized objects present in the aforementioned time-series images, Based on at least one of the point cloud datasets, the bounding box of the determined unrecognized object is determined. The second neural network (138) obtains a neural network representation of the unrecognized object based on the time-series images and the determined bounding box. Based on the acquired neural network representation of the unrecognized object, the first neural network (132) is trained for the recognition of the unrecognized object. A processing unit (130) configured as follows and A visual-based perception system (100) comprising the above.

2. The processing unit (130) is To obtain each point cluster of the aforementioned point cloud dataset, each point in the aforementioned point cloud dataset is clustered, The unrecognized objects are determined by determining, with respect to at least a portion of the point cloud dataset, that one of the point clusters does not correspond to any of the objects recognized by the first neural network (132). A visual-based perception system (100) according to claim 1, configured as follows.

3. The processing unit (130) is To determine the ground, ground segmentation is performed based on the point cloud dataset. The unrecognized object is determined by determining that the unrecognized object is located on the determined ground surface. A visual-based perception system (100) according to claim 2, configured as follows.

4. The processing unit (130) is With respect to each of the multiple pre-stored images captured by the camera device (120) and the corresponding pre-stored LIDAR point cloud datasets captured by the LIDAR device (110), at least one other object having the corresponding bounding box determined by the processing unit (130) is determined based on at least one of the corresponding pre-stored LIDAR point cloud datasets having the same dimensions as the bounding box determined by the processing unit (130) with respect to the determined unrecognized object, within a predetermined threshold. To obtain a plurality of training images for training the first neural network (132) for the recognition of the unrecognized object, in each of the plurality of pre-stored images, replace the determined at least one other object with the neural network representation of the unrecognized object. It is further configured in the following way: A visual-based perception system (100) according to any one of claims 1 to 3.

5. The processing unit (130) is With respect to each of the plurality of pre-stored images captured by the camera device (120), a first direction of light relative to at least one other object is determined, and a second direction of light relative to the determined unrecognized object is determined. When the direction of the first light and the direction of the second light are offset from each other by less than a predetermined illumination angle, the determined at least one other object in each of the plurality of pre-stored images is replaced with the neural network representation of the unrecognized object. A visual-based perception system (100) according to claim 4, configured as follows.

6. The second neural network (138) includes a first multilayer perceptron MLP trained on neural radiance field technology and a second MLP different from the first MLP, also trained on neural radiance field technology. The first MLP is configured to acquire the neural network representation of the unrecognized object, and the second MLP is configured to acquire the neural network representation of the background of the unrecognized object. The visual-based perception system (100) according to claim 4.

7. The first MLP is configured to acquire the neural network representation of the unrecognized object based on the portion of the camera ray that crosses the region corresponding to the bounding box. The second MLP is configured to acquire the neural network representation of the background of the unrecognized object based on the portion of the camera ray that does not cross the region corresponding to the bounding box. The visual-based perception system (100) according to claim 6.

8. The first MLP is configured to acquire the neural network representation of the unrecognized object based on the camera orientation. The processing unit (130) is configured to replace at least one other determined object with the neural network representation of the unrecognized object by rendering the neural network representation of the unrecognized object onto each of the plurality of pre-stored images based on a rendering pose that deviates from the camera pose by less than a predetermined threshold, The visual-based perception system (100) according to claim 6.

9. The visual-based perception system (100) according to claim 1, wherein the visual-based perception system (100) is configured to be installed on a vehicle, and the time-series point cloud dataset and the time-series images represent a driving scene of the vehicle.

10. A method (200) for training a first neural network (132) of a visual-based perception system (100) for object recognition, Step (S210) of acquiring a time-series point cloud dataset representing the environment of the visual-based perception system (100) using the light detection and ranging LIDAR device of the visual-based perception system (100), The steps include: capturing a time-series image of the environment using the camera device (120) of the visual-based perception system (100) (S220), The process includes a step (S230) in which the processing unit (130) of the visual-based perception system (100) determines an unrecognized object in the time-series image, The processing unit (130) performs the step (S240) of determining the bounding box of the determined unrecognized object based on at least one of the point cloud datasets, Step (S250) of obtaining a neural network representation of the unrecognized object based on the time-series image and the determined bounding box by a second neural network (138) of the visual-based perception system (100), Step (S260) of training the first neural network (132) for the recognition of the unrecognized object based on the acquired neural network representation of the unrecognized object, A method including (200).

11. The processing unit (130) performs the step of clustering each point in the point cloud dataset in order to obtain each point cluster in the point cloud dataset. It further includes, The unrecognized objects are determined by determining, with respect to at least a portion of the point cloud dataset, that one of the point clusters does not correspond to any of the objects recognized by the first neural network (132) (S230). The method according to claim 10 (200).

12. The processing unit (130) performs a step of performing ground segmentation based on the point cloud dataset in order to determine the ground. It further includes, The unrecognized object is determined by determining that the unrecognized object is located on the determined ground (S230). The method according to claim 11 (200).

13. The processing unit (130) determines, with respect to each of a plurality of pre-stored images captured by the camera device (120) and a corresponding pre-stored LIDAR point cloud dataset captured by the LIDAR device (110), at least one other object having the same dimensions as the bounding box determined by the processing unit (130) for the determined unrecognized object, based on at least one of the corresponding pre-stored LIDAR point cloud datasets having the same dimensions as the bounding box determined by the processing unit (130) for the determined unrecognized object, within a predetermined threshold; To obtain a plurality of training images for training the first neural network (132) for the recognition of the unrecognized object, the steps include replacing the determined at least one other object in each of the plurality of pre-stored images with the neural network representation of the unrecognized object. The method according to claim 10 (200), further comprising:

14. The processing unit (130) determines a first light direction toward at least one other object with respect to each of the plurality of pre-stored images captured by the camera device (120), and determines a second light direction toward the determined unrecognized object. It further includes, The method according to claim 13 (200), wherein the at least one other object determined is replaced in each of the plurality of pre-stored images by the neural network representation of the unrecognized object when the first light direction and the second light direction are offset from each other by less than a predetermined illumination angle.

15. The second neural network includes a first multilayer perceptron MLP trained on neural radiance field technology and a second MLP different from the first MLP, also trained on neural radiance field technology. The neural network representation of the unrecognized object is obtained by the first MLP (S250), The second MLP obtains a neural network representation of the background of the unrecognized object. The method according to claim 13, further comprising (200).

16. The neural network representation of the unrecognized object is obtained based on the portion of the camera ray that crosses the region corresponding to the bounding box (S250), The neural network representation of the background of the unrecognized object is obtained based on the portion of the camera ray that does not cross the region corresponding to the bounding box. The method according to claim 15 (200).

17. The neural network representation of the unrecognized object is obtained based on the camera pose (S250), The at least one other object determined above is replaced by the neural network representation of the unrecognized object by rendering the neural network representation of the unrecognized object onto each of the plurality of pre-stored images based on a rendering pose that deviates from the camera pose by less than a predetermined threshold. The method according to claim 15 (200).

18. A computer program, when executed on a computer, comprising computer-readable instructions for performing each step of the method (200) according to any one of claims 10 to 17.

Images