Computational efficient unsupervised dnn pre-training

Abstract

The present invention relates to computationally efficient unsupervised DNN pre-training. A system includes a computer comprising a processor and a memory. The memory includes instructions that cause the processor to be programmed to determine, based on features extracted from a first cropped image portion and corresponding point cloud data and features extracted from a second cropped image portion and corresponding point cloud data, a pair-wise region of interest feature similarity. The processor is further programmed to determine, based on the pair-wise region of interest feature similarity, a loss using a loss function, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network. The processor is further programmed to update at least one weight of at least one of the first deep neural network or the second deep neural network based on the loss.

Description

Technical Field

[0001] This disclosure relates to the use of self-supervised techniques to pre-train deep neural networks. Background Technology

[0002] Various sensors can provide data to identify objects in the physical (i.e., real-world) environment surrounding a vehicle. For example, sensors such as cameras, lidar, radar, and ultrasonic sensors can provide data to identify objects. The vehicle's computer, operating the vehicle in this environment, can receive data from these sensors. Summary of the Invention

[0003] A system includes a computer comprising a processor and a memory. The memory includes instructions that cause the processor to determine pairwise region-of-interest (ROI) feature similarity based on features extracted from a first cropped image portion and corresponding point cloud data, and features extracted from a second cropped image portion and corresponding point cloud data. The processor is also programmed to determine a loss based on the pairwise ROI feature similarity using a loss function, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network. The processor is further programmed to update at least one weight of at least one of the first deep neural network or the second deep neural network based on the loss.

[0004] Among other features, the processor is also programmed to receive images and corresponding point cloud data.

[0005] Among other features, the processor is also programmed to perform a random cropping function on the image and the corresponding point cloud data to generate a first cropped image portion and a second cropped image portion.

[0006] Among other features, the image includes two-dimensional images.

[0007] Among other features, two-dimensional images include red-green-blue (RGB) images.

[0008] Among other features, the first deep neural network includes a two-dimensional convolutional neural network.

[0009] Among other features, the second deep neural network includes a three-dimensional convolutional neural network.

[0010] The vehicle may include a computer. The computer includes a processor and a memory, and the memory includes instructions that cause the processor to determine pairwise region-of-interest (ROI) feature similarity based on features extracted from a first cropped image portion and corresponding point cloud data, and features extracted from a second cropped image portion and corresponding point cloud data. The processor is also programmed to determine a loss based on the pairwise RIO feature similarity using a loss function, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network. The processor is further programmed to update at least one weight of at least one of the first deep neural network or the second deep neural network based on the loss.

[0011] Among other features, the processor is also programmed to receive images and corresponding point cloud data.

[0012] Among other features, the processor is also programmed to perform a random cropping function on the image and the corresponding point cloud data to generate a first cropped image portion and a second cropped image portion.

[0013] Among other features, the image includes two-dimensional images.

[0014] Among other features, two-dimensional images include red-green-blue (RGB) images.

[0015] Among other features, the first deep neural network includes a two-dimensional convolutional neural network.

[0016] Among other features, the second deep neural network includes a three-dimensional convolutional neural network.

[0017] Based on features extracted from a first cropped image portion and corresponding point cloud data, and features extracted from a second cropped image portion and corresponding point cloud data, a pair of region-of-interest (ROI) feature similarities are determined. The method further includes: determining a loss based on the pair of ROI feature similarities using a loss function, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network; and updating at least one weight of at least one of the first deep neural network or the second deep neural network based on the loss.

[0018] Among other features, the method includes receiving an image and the corresponding point cloud data.

[0019] Among other features, the method includes performing a random cropping function on the image and the corresponding point cloud data to generate a first cropped image portion and a second cropped image portion.

[0020] Among other features, the image includes two-dimensional images.

[0021] Among other features, the first deep neural network includes a two-dimensional convolutional neural network.

[0022] Among other features, the second deep neural network includes a three-dimensional convolutional neural network.

[0023] Further applicability will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0024] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0025] Figure 1 It is a block diagram of an example system including a vehicle;

[0026] Figure 2 This is a block diagram of an example computing device;

[0027] Figure 3 It is a diagram illustrating an example deep neural network;

[0028] Figure 4 It is a block diagram illustrating an example feature extractor; and

[0029] Figure 5 This is a flowchart illustrating an example process of using self-supervised techniques to pre-train a feature extractor. Detailed Implementation

[0030] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0031] Typically, labeled training data can be used to train neural networks, such as convolutional neural networks. Labeled training data can include images with bounding boxes that include one or more labeled objects to be detected by an object detector. However, annotating training data to include bounding boxes and labels, or even image labels as in weakly supervised object detection, can be resource-intensive and time-consuming. This disclosure provides a self-supervised learning system that eliminates the need for annotated training data.

[0032] Figure 1 This is a block diagram of example system 100. System 100 includes a vehicle 105, which may include land vehicles such as cars and trucks, aircraft, and / or water vehicles. Vehicle 105 includes a computer 110, vehicle sensors 115, actuators 120 for actuating various vehicle components 125, and a vehicle communication module 130. Communication module 130 allows computer 110 to communicate with server 145 via network 135.

[0033] Computer 110 can operate vehicle 105 in automatic, semi-automatic, or non-automatic (manual) modes. For the purposes of this disclosure, automatic mode is defined as a mode in which each of the propulsion, braking, and steering of vehicle 105 is controlled by computer 110; in semi-automatic mode, computer 110 controls one or both of the propulsion, braking, and steering of vehicle 105; and in non-automatic mode, a human operator controls each of the propulsion, braking, and steering of vehicle 105.

[0034] Computer 110 may include programming to operate one or more of the following functions of vehicle 105: braking, propulsion (e.g., controlling the vehicle's acceleration by controlling one or more of an internal combustion engine, electric motor, hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, and to determine whether and when computer 110 (as opposed to a human operator) controls these operations. Additionally, computer 110 may be programmed to determine whether and when a human operator controls these operations.

[0035] Computer 110 may include one or more processors (e.g., included in an Electronic Controller Unit (ECU) or similar unit within vehicle 105 for monitoring and / or controlling various vehicle components 125, such as powertrain controllers, brake controllers, steering controllers, etc.) or be communicatively coupled to one or more processors via vehicle 105 communication module 130 as further described below. Further, computer 110 may communicate with a navigation system using a Global Positioning System (GPS) via vehicle 105 communication module 130. As an example, computer 110 may request and receive location data from vehicle 105. The location data may be in a known form, such as geographic coordinates (latitude and longitude coordinates).

[0036] Computer 110 is typically configured to communicate on communication module 130 of vehicle 105, and also to wired and / or wireless networks (such as buses in vehicle 105, such as Controller Area Network (CAN)) and / or other wired and / or wireless mechanisms within vehicle 105.

[0037] Via the vehicle 105 communication network, the computer 110 can send and / or receive messages from various devices within the vehicle 105, such as vehicle sensors 115, actuators 120, vehicle components 125, human-machine interfaces (HMIs), etc. Alternatively or additionally, where the computer 110 actually comprises multiple devices, the vehicle 105 communication network can be used for communication between devices represented herein as computer 110. Further, as mentioned below, various controllers and / or vehicle sensors 115 can provide data to the computer 110. The vehicle 105 communication network may include one or more gateway modules that provide interoperability between various networks and devices within the vehicle 105, such as protocol converters, impedance matching devices, rate converters, etc.

[0038] Vehicle sensors 115 may include various devices, such as those known to provide data to computer 110. For example, vehicle sensors 115 may include light detection and ranging (LiDAR) sensors 115 disposed on top of vehicle 105, behind the windshield of vehicle 105, or around vehicle 105, providing information on the relative position, size, and shape of objects and / or the conditions around vehicle 105. As another example, one or more radar sensors 115 fixed to the bumper of vehicle 105 may provide data to provide distance, speed, etc., of objects or the like relative to the position of vehicle 105. Vehicle sensors 115 may also include camera sensors 115, such as forward-looking, side-looking, and rear-looking sensors, to provide images of views from inside and / or outside vehicle 105.

[0039] The actuator 120 of the vehicle 105 is implemented via circuits, chips, motors, or other electronic and / or mechanical components that can actuate various vehicle subsystems according to known and appropriate control signals. The actuator 120 can be used to control the component 125, including braking, accelerating, and steering the vehicle 105.

[0040] In the context of this disclosure, vehicle component 125 is one or more hardware components adapted to perform mechanical or electromechanical functions or operations (e.g., moving vehicle 105, slowing or stopping vehicle 105, steering vehicle 105, etc.). Non-limiting examples of component 125 include propulsion components (which include, for example, internal combustion engines and / or electric motors), transmission components, steering components (which may include, for example, one or more of a steering wheel, steering rack, etc.), braking components (as described below), parking assist components, adaptive cruise control components, adaptive steering components, movable seats, etc.

[0041] Additionally, computer 110 can be configured to communicate with devices external to vehicle 105 via vehicle-to-vehicle communication module or interface 130, for example, via vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) wireless communication to another vehicle, to a remote server 145 (such as an edge server) (typically via network 135). Module 130 may include one or more mechanisms that computer 110 can use for communication, including wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or topology when using multiple communication mechanisms) and any desired combination thereof. Exemplary communications provided via module 130 include cellular, IEEE 802.11, Dedicated Short-Range Communication (DSRC), and / or Wide Area Networks (WAN) (including the Internet that provides data communication services).

[0042] Network 135 can be one or more of various wired or wireless communication mechanisms, including wired (e.g., cable and fiber optic) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or topology when using multiple communication mechanisms) and any desired combination thereof. Exemplary communication networks include wireless communication networks (e.g., using Bluetooth, Bluetooth Low Energy (BLE), IEEE 802.11, vehicle-to-vehicle (V2V) communication such as Dedicated Short Range Communication (DSRC), etc.), local area networks (LANs), and / or wide area networks (WANs) (including the Internet) to provide data communication services.

[0043] Figure 2 An example computing device 200, namely a computer 110 and a server 145, is illustrated in the figure, configured to perform one or more processes described herein. As shown, the computing device may include a processor 205, a memory 210, a storage device 215, an I / O interface 220, and a communication interface 225. Furthermore, the computing device 200 may include input devices such as a touchscreen, a mouse, and a keyboard. In some embodiments, the computing device 200 may include a... Figure 2 The number of components shown is less or more.

[0044] In a particular implementation, processor 205 includes hardware for executing instructions, such as those that constitute a computer program. By way of example and not limitation, in order to execute instructions, processor 205 may retrieve (or fetch) instructions from internal registers, internal cache, memory 210, or storage device 215, and decode and execute those instructions.

[0045] Computing device 200 includes memory 210 coupled to processor 205. Memory 210 can be used to store data, metadata, and programs executed by the processor. Memory 210 may include one or more volatile and non-volatile memories, such as random-access memory ("RAM"), read-only memory ("ROM"), solid-state disk ("SSD"), flash memory, phase-change memory ("PCM"), or other types of data storage. Memory 210 may be internal memory or distributed memory.

[0046] Computing device 200 includes storage device 215, which includes storage means for storing data or instructions. By way of example and not limitation, storage device 215 may include the non-transitory storage media described above. Storage device 215 may include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0047] The computing device 200 also includes one or more input or output (“I / O”) devices / interfaces 220, providing these devices / interfaces to allow a user to provide input (such as user strokes) to the computing device 200, receive output from the computing device 200, and otherwise transmit data to and from the computing device 200. These I / O devices / interfaces 220 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or combinations of such I / O devices / interfaces 220. The touchscreen can be activated with a writing device or a finger.

[0048] I / O device / interface 220 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, device / interface 220 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular embodiment.

[0049] The computing device 200 may also include a communication interface 225. The communication interface 225 may include hardware, software, or both. The communication interface 225 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices 200 or one or more networks. By way of example and not limitation, the communication interface 225 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired networks, or a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks such as Wi-Fi. The computing device 200 may also include a bus 230. The bus 230 may include hardware, software, or both for coupling components of the computing device 200 to each other.

[0050] Figure 3 This is a diagram of an example Deep Neural Network (DNN) 300. DNN 300 can represent the neural network described in this paper. DNN 300 includes multiple nodes 305, and the nodes 305 are arranged such that DNN 300 includes an input layer, one or more hidden layers, and an output layer. Each layer of DNN 400 may include multiple nodes 305. Although Figure 3 The diagram illustrates three (3) hidden layers, but it should be understood that a DNN 300 may include more or fewer hidden layers. The input and output layers may also include one (1) or more nodes 305.

[0051] Node 305 is sometimes referred to as artificial neuron 305 because it is designed to mimic biological neurons (such as human neurons). Each neuron 305 has a set of inputs (indicated by arrows) multiplied by its respective weights. The weighted inputs are then summed in an input function to provide (possibly adjusted for bias) a net input. This net input is then provided to an activation function, which in turn provides the output to the connected neurons 305. The activation function can be a variety of suitable functions, typically chosen based on empirical analysis. For example... Figure 3As indicated by the arrow in the diagram, the output of neuron 305 can then be included in a set of inputs to one or more neurons 305 in the next layer.

[0052] The DNN 300 can be trained to accept data as input and generate output based on that input. Ground-based data (i.e., data about real-world conditions or states) can be used to train the DNN 300. For example, a processor can train the DNN 300 using ground-based data or update it using additional data. For example, the weights can be initialized using a Gaussian distribution, and the bias of each node 305 can be set to zero. Training the DNN 300 can include updating the weights and biases via appropriate techniques such as optimized backpropagation. Ground-based data can include, but is not limited to, data used for classifying and / or recognizing objects in a point cloud.

[0053] As discussed in this paper, DNN 300 can include Convolutional Neural Networks (CNNs). CNNs are a class of deep feedforward artificial neural networks that can utilize variations of multilayer perceptrons designed to require minimal preprocessing. Through training, CNNs determine filters that can be used to identify features of interest, regions of interest, etc. In other words, CNNs can extract features from input data using one or more filters.

[0054] In various implementations, the DNN 300 may include a two-dimensional (2D) CNN and / or a three-dimensional (3D) CNN that performs feature extraction. In this context, temporal and / or point cloud data may be fed to the 3D CNN, and spatial data may be fed to the 2D CNN, as discussed in further detail below.

[0055] Figure 4 An example feature extractor 400 according to an exemplary implementation of this disclosure is illustrated. As discussed herein, the feature extractor 400 is pre-trained using images and corresponding point cloud data. The images and point cloud data do not include labels, masks, bounding boxes, etc. It should be understood that the masks and / or bounding boxes are not necessarily manually annotated.

[0056] As shown in the figure, the feature extractor 400 may include a first convolutional neural network (CNN) 405, a second convolutional neural network (CNN) 410, and a feature fusion module 425. In this embodiment, the first CNN 405 includes a 2D CNN, and the second CNN 410 includes a 3D CNN.

[0057] Each CNN 405, CNN 410 receives sensor data captured by sensor 115. The sensor data may include image data 415 and point cloud data 420 captured by sensor 115. In an example implementation, image data 415 may include red-green-blue (RGB) color images captured by camera sensor 115, and point cloud data 420 may include a dataset created when LiDAR and / or radar sensor 115 scans an area.

[0058] CNN 405 and CNN 410 each output feature maps based on their respective inputs. Each feature map is provided as input to feature fusion module 425. Feature fusion module 425 can fuse features from the received feature maps. Feature fusion module 425 can use various fusion techniques to fuse features with each other. For example, fusion techniques may include, but are not limited to, competitive fusion techniques, complementary sensor techniques, and / or cooperative fusion techniques. Feature fusion module 425 can output data representing the fused features to computer 110 for further processing. For example, computer 110 may include additional software modules providing segmentation, detection, classification, tracking, and other functions.

[0059] Figure 5 This is a flowchart of an exemplary process 500 that uses a self-supervised technique to pre-train a feature extractor (such as feature extractor 400). The boxes of process 500 can be executed by computer 110 or server 145. Process 500 begins at box 505, where it is determined whether data has been received. In an example implementation, the data may include an image, such as a two-dimensional (2D) RGB image and corresponding point cloud data. If no data is received, process 500 returns to box 505.

[0060] Otherwise, at box 510, one or more Regions of Interest (ROIs) are determined within the image and the corresponding point cloud data. In an example implementation, a first CNN 405 can determine the ROIs within the image data, and a second CNN 410 can determine the ROIs within the point cloud data. In other example implementations, the ROIs in the image can be computed using image processing algorithms, such as active contours or region growing algorithms. Since both the image and the corresponding point cloud data are available, the initial seed for the region growing superpixel segmentation algorithm can be initialized using the point cloud projected into 2D image space. This region growing superpixel segmentation algorithm can then segment the image into potential ROIs.

[0061] Once the ROI in the image is calculated, the corresponding frustum is extracted from the point cloud. Then, the image ROI features are cropped from the 2D features, and the point cloud frustum features are cropped from the 3D features.

[0062] At box 515, at least two (2) random cropping processes are performed on the image data and the corresponding point cloud data. The cropping processes may produce at least two different views within the image data and the corresponding point cloud data. At box 520, the region of interest correspondence is determined based on the regions of interest from the uncropped data, the cropped data corresponding to the first crop, and the cropped data corresponding to the second crop.

[0063] At box 525, the image and point cloud features corresponding to the first crop are determined. At box 530, the image and point cloud features corresponding to the second crop are determined. It should be understood that boxes 525 and 530 can occur before box 520.

[0064] At box 535, the region of interest (ROI) features corresponding to the first cropping are calculated. At box 540, the ROI features corresponding to the second cropping are calculated. At box 545, computer 110 and / or server 145 apply appropriate projection and prediction functions to the ROI features corresponding to the first cropping, i.e., features within the ROI region corresponding to the first cropping. At box 550, computer 110 and / or server 145 apply appropriate projection and prediction functions to the ROI features corresponding to the second cropping, i.e., features within the ROI region corresponding to the second cropping.

[0065] At box 555, computer 110 and / or server 145 calculates pairwise region-of-interest (ROI) feature similarity. In an example implementation, computer 110 and / or server 145 uses a logarithmic softmax function applied to the feature dot product calculation to calculate the pairwise ROI feature similarity. The feature dot product calculation can be determined using appropriate matrix multiplication applied to the ROI correspondence and features within the ROI regions of the first and second clippings.

[0066] At box 560, computer 110 and / or server 145 calculate the loss based on the loss function of feature extractor 400. At box 565, based on the calculated loss, one or more weights and / or parameters of the first CNN 405 and the second CNN 410 are updated, the projection function is updated, and / or the prediction function is updated. Appropriate backpropagation techniques can be used to update the weights and / or parameters. In an example implementation, the weights and / or parameters of the first CNN 405 can be updated using backpropagation, while the weights of the second CNN 410 can be updated using backpropagation or as an exponential moving average of the updated weights of the first CNN 405. Process 500 then ends. Once trained using the self-supervised training process described herein, feature extractor 400 can be used by computer 110 to operate vehicle 105 in an automatic or semi-automatic manner.

[0067] The description in this disclosure is merely exemplary in nature, and any changes that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such changes should not be considered as departing from the spirit and scope of this disclosure.

[0068] Typically, the described computing system and / or device may employ any of a variety of computer operating systems, including, but not limited to, the following versions and / or variants: Microsoft Operating system, Microsoft Operating systems, Unix operating systems (for example, those released by Oracle Corporation, located in Redwood City, California). Operating systems include: AIX UNIX (published by International Business Machines in Armonk, New York), Linux; Mac OSX and iOS (published by Apple Inc. in Cupertino, California); BlackBerry OS (published by BlackBerry, Ltd. in Waterloo, Canada); Android (developed by Google and the Open Handset Alliance); and infotainment systems provided by QNX software systems. CAR platform. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktop computers, laptops, handheld computers, or other computing systems and / or devices.

[0069] Computers and computing devices typically include computer-executable instructions, which can be executed by one or more computing devices (such as those listed above). Computer-executable instructions can be compiled or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, alone or in combination. TM C, C++, Matlab, Simulink, StateFlow, Visual Basic, JavaScript, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine and the Dalvik Virtual Machine. Typically, a processor (e.g., a microprocessor) receives instructions from memory, computer-readable media, etc., and executes those instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transferred using various computer-readable media. Files in computing devices are typically collections of data stored on computer-readable media such as storage media and random access memory.

[0070] Memory can include computer-readable media (also known as processor-readable media), which includes any non-transitory (e.g., tangible) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Non-volatile media can include, for example, optical discs or magnetic disks, and other permanent storage devices. Volatile media can include, for example, dynamic random-access memory (DRAM), which typically constitutes main memory. Such instructions can be transmitted via one or more transmission media, including coaxial cables, copper wires, and optical fibers, including wires containing the system bus coupled to the processor of the ECU. Common forms of computer-readable media include, for example, floppy disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cassette tape, or any other computer-readable medium.

[0071] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing, and retrieving a wide range of data, including hierarchical databases, a set of files in a file system, application databases in proprietary formats, Relational Database Management Systems (RDBMS), and so on. Each such data store is typically contained on a computing device employing a computer operating system such as one of the computer operating systems mentioned above, and is accessed via a network in any one or more of various ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used for creating, storing, editing, and executing stored programs, RDBMS typically employs a Structured Query Language (SQL), such as PL / SQL mentioned above.

[0072] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on an associated computer-readable medium (e.g., disks, storage, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0073] In this application, the term "module" or "controller" may be replaced by the term "circuit" as defined below. The term "module" may refer to, be part of, or include the following components: Application Specific Integrated Circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuits; digital, analog, or mixed-signal analog / digital integrated circuits; combinational logic circuits; Field Programmable Gate Array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0074] A module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of this disclosure may be distributed across multiple modules connected via the interface circuit. For example, multiple modules may enable load balancing. In another example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0075] Regarding the media, processes, systems, methods, and inspirations described herein, it should be understood that although the steps of such processes are described as occurring according to an ordered sequence, such processes can be practiced using steps performed in a different order than those described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should not be construed as limiting the claims in any way.

[0076] Therefore, it should be understood that the above description is illustrative and not restrictive. Many embodiments and applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. The scope of the invention should not be determined by reference to the above description, but rather by reference to the appended claims and the full scope of their equivalents. It is foreseeable and anticipated that future developments will occur in the art discussed herein, and that the disclosed systems and methods will be incorporated into such future implementations. In conclusion, it should be understood that the invention is capable of modifications and variations and is limited only by the appended claims.

[0077] All terms used in the claims are intended to be given their generic and ordinary meaning as understood by those skilled in the art, unless expressly indicated otherwise herein. In particular, the use of singular articles such as “a”, “the”, “said”, etc., should be interpreted as referring to one or more of the indicated elements, unless the claims expressly limit them to the contrary.

Claims

1. A system comprising a computer, the computer including a processor and a memory, the memory including instructions that program the processor to: Receive an image and corresponding point cloud data, wherein the image and the point cloud data do not include labels, masks and / or bounding boxes; The projection function and prediction function are applied to the features and corresponding point cloud data extracted from the first cropped image portion and the features and corresponding point cloud data extracted from the second cropped image portion; Based on the features applied by the projection function and the prediction function, the similarity of paired regions of interest features is determined; Based on the similarity of the paired regions of interest features, a loss function is used to determine the loss, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network; as well as Based on the loss, update at least one weight of at least one of the first deep neural network or the second deep neural network; The first deep neural network is configured to process the image data, and the second deep neural network is configured to process the point cloud data.

2. The system according to claim 1, wherein, The processor is also programmed to perform a random cropping function on the image and the corresponding point cloud data to generate the first cropped image portion and the second cropped image portion.

3. The system according to claim 1, wherein, The image includes a two-dimensional image.

4. The system according to claim 3, wherein, The two-dimensional image includes a red-green-blue (RGB) image.

5. The system according to claim 1, wherein, The first deep neural network includes a two-dimensional convolutional neural network.

6. The system according to claim 5, wherein, The second deep neural network includes a three-dimensional convolutional neural network.

7. A vehicle comprising a computer, the computer including a processor and a memory, the memory including instructions that program the processor to: Receive an image and corresponding point cloud data, wherein the image and the point cloud data do not include labels, masks and / or bounding boxes; The projection function and prediction function are applied to the features and corresponding point cloud data extracted from the first cropped image portion and the features and corresponding point cloud data extracted from the second cropped image portion; Based on the features applied by the projection function and the prediction function, the similarity of paired regions of interest features is determined; Based on the similarity of the paired regions of interest features, a loss function is used to determine the loss, wherein the loss function corresponds to at least one of a first deep neural network or a second deep neural network; as well as Based on the loss, update at least one weight of at least one of the first deep neural network or the second deep neural network; The first deep neural network is configured to process the image data, and the second deep neural network is configured to process the point cloud data.

8. The vehicle according to claim 7, wherein, The processor is also programmed to perform a random cropping function on the image and the corresponding point cloud data to generate the first cropped image portion and the second cropped image portion.

Images