Systems and methods for classifying objects detected in images at an autonomous driving system

Abstract

An autonomous driving system includes an object detection system. A neural network image encoder generates an image embedding associated with an image including an object. A neural network text encoder generates a concept embedding associated with each of a plurality of concepts. Each of the plurality of concepts is associated with one of at least two object classes. A confidence score module generates a confidence score for each of the plurality of concepts based on the image embedding and the concept embedding associated with the concept. An object class prediction module generates a predicted object class for the object based on an association between a concept set of the plurality of concepts having at least two of the highest values of the generated confidence scores and one of the at least two object classes associated with a majority of the concept set.

Description

TECHNICAL FIELD

[0001] The technical field generally relates to autonomous vehicles, and more particularly to systems and methods for classifying objects detected in images at an automated driving system (ADS). BACKGROUND

[0002] An ADS of an autonomous vehicle typically receives images from a vehicle sensor system. The received images typically represent observable conditions of an external environment of the autonomous vehicle. The ADS can rely on classification of objects detected in the images to manage performance of the autonomous vehicle.

[0003] Images captured by a vehicle sensor system in a relatively low-visibility environment (e.g., during inclement weather conditions) can not be as clear as images captured during clear weather conditions. Representative features typically used to classify objects detected in images can not be as easily discernible in such images. It would be beneficial to use an object detection system that is capable of classifying objects detected in images captured in low-visibility environments. SUMMARY

[0004] In one embodiment, an automated driving system (ADS) of an autonomous vehicle includes an object detection system. The object detection system includes: a neural network image encoder configured to generate an image embedding associated with at least a portion of an image generated by a vehicle sensor system, the at least a portion of the image including an object; a neural network text encoder configured to generate a concept embedding associated with each of a plurality of concepts, each of the plurality of concepts being associated with one of at least two object classes; a confidence score module configured to generate a confidence score for each of the plurality of concepts based on the image embedding and the concept embedding associated with the concept; an object class prediction module configured to generate a predicted object class of the object based on an association between a concept set of the plurality of concepts having at least two of the highest values of the generated confidence scores and one of the at least two object classes associated with a majority of the concept set; and a processor configured to manage performance of the autonomous vehicle based in part on the predicted object class of the object.

[0005] In one embodiment, each of the plurality of concepts is associated with one of the at least two object classes including a predefined concept.

[0006] In one embodiment, the object class of the at least two object classes includes at least two of a sedan, a bus, a bicycle, a motorcycle, and a pedestrian.

[0007] In one embodiment, the system further includes an object candidate module configured to detect the object in an image received from a vehicle sensor system and generate at least a portion of the image based on a location of the detected object in the image.

[0008] In one embodiment, the object candidate module is configured to implement one of a region-based Fast R-CNN algorithm and a You Only Look Once (YOLO) method.

[0009] In one embodiment, the system further includes a feature extraction neural network including a neural network image encoder and a neural network text encoder, wherein the feature extraction neural network is configured to be pre-trained to connect, by a feature extraction neural network training system, a plurality of concepts and image data associated with each of the at least two object classes.

[0010] In one embodiment, the confidence score module is configured to generate the confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embedding and the concept embedding associated with the concept.

[0011] In one embodiment, a computer-readable medium includes instructions stored thereon for classifying an object detected in an image at an ADS, which when executed by a processor, cause the processor to: generate an image embedding associated with at least a portion of an image generated by a vehicle sensor system, the at least a portion of the image including the object; generate a concept embedding associated with each of a plurality of concepts, each of the plurality of concepts being associated with one of at least two object classes; generate a confidence score for each of the plurality of concepts based on the image embedding and the concept embedding associated with the concept; generate a predicted object class of the object based on an association between a concept set of the plurality of concepts having at least two of a highest value of the generated confidence scores and one of the at least two object classes associated with a majority of the concept set; and manage a performance of an autonomous vehicle based in part on the predicted object class of the object.

[0012] In one embodiment, the computer-readable medium further includes instructions that cause the processor to generate the concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes including a predefined concept.

[0013] In one embodiment, the computer-readable medium further includes instructions that cause the processor to generate the concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes including at least two of a car, a bus, a bicycle, a motorcycle, and a pedestrian.

[0014] In one embodiment, the computer-readable medium further includes instructions that cause the processor to detect the object in an image received from the vehicle sensor system and generate at least a portion of the image based on a location of the detected object in the image.

[0015] In one embodiment, the computer-readable medium further includes instructions that cause the processor to detect the object in an image received from the vehicle sensor system using one of a region-based Fast R-CNN algorithm and a You Only Look Once (YOLO) method and generate at least a portion of the image based on a location of the detected object in the image.

[0016] In one embodiment, the computer-readable medium further includes instructions that cause the processor to pre-train a feature extraction neural network to connect a plurality of concepts and image data associated with each of at least two object classes, the feature extraction neural network including a neural network image encoder configured to generate an image embedding associated with at least a portion of an image including an object and a neural network text encoder configured to generate a concept embedding associated with each of the plurality of concepts.

[0017] In one embodiment, the computer-readable medium further includes instructions that cause the processor to generate a confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embedding and the concept embedding associated with the concept.

[0018] In one embodiment, a method for classifying an object in an image at an autonomous driving system (ADS) of an autonomous vehicle includes generating, at a neural network image encoder, an image embedding associated with at least a portion of an image generated by a vehicle sensor system, the at least a portion of the image including the object; generating, at a neural network text encoder, a concept embedding associated with each of a plurality of concepts, each of the plurality of concepts being associated with one of at least two object classes; generating, at a confidence score module, a confidence score for each of the plurality of concepts based on the image embedding and the concept embedding associated with the concept; generating, at an object class prediction module, a predicted object class of the object based on an association between a concept set of the plurality of concepts having at least two of the highest values of the generated confidence scores and one of the at least two object classes associated with a majority of the concept set; and managing, at a controller, a performance of the autonomous vehicle based in part on the predicted object class of the object.

[0019] In one embodiment, the method further includes generating the concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes including a predefined concept.

[0020] In one embodiment, the method further includes generating a concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes, each of the plurality of concepts being associated with one of the at least two object classes including at least two of a car, a bus, a bicycle, a motorcycle, and a pedestrian.

[0021] In one embodiment, the method further includes detecting an object in an image received from a vehicle sensor system; and generating at least a portion of the image based on a location of the detected object in the image.

[0022] In one embodiment, the method further includes pre-training a feature extraction neural network including a neural network image encoder and a neural network text encoder, thereby concatenating, at a feature extraction neural network training system, the plurality of concepts and image data associated with each of the at least two object classes.

[0023] In one embodiment, the method further includes generating a confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embedding and the concept embedding associated with the concept. BRIEF DESCRIPTION OF DRAWINGS

[0024] In the following, exemplary embodiments will be described with reference to the following drawings, wherein like numbers denote like elements.

[0025] Figure 1 is a functional block diagram representing an autonomous vehicle including an embodiment of an object detection system;

[0026] Figure 2 is a functional block diagram representing an embodiment of a feature extraction neural network training system configured to train an embodiment of a feature extraction neural network;

[0027] Figure 3 is a flowchart representing an example of a method of training an embodiment of a feature extraction neural network;

[0028] Figure 4 is a functional block diagram of an embodiment of an object detection system including a feature extraction neural network;

[0029] Figure 5 is a flowchart representing an example of a method of classifying objects detected in an image using an embodiment of an object detection system at an autonomous driving system;

[0030] Figure 6A and Figure 6B is a table including examples of confidence scores associated with example concepts of a car, a bus, a bicycle, a motorcycle, and a person generated by a confidence score module of an embodiment of an object detection system; and

[0031] Figure 7 This is a flowchart illustrating an example of a method for classifying objects detected in an image using an object detection system in an autonomous driving system. Detailed Implementation

[0032] The following detailed description is merely exemplary in nature and is not intended to limit the application and use of this application. Furthermore, it is not intended to be bound by any express or implied theory set forth in the foregoing introduction, summary of the invention, or the following detailed description. As used herein, the term "module" refers to any hardware, software, firmware, electronic control components, processing logic, and / or processor device, individually or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the described functionality.

[0033] The embodiments of this disclosure may be described herein as functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, or lookup tables, which may perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be practiced in conjunction with any number of systems, and the systems described herein are merely exemplary embodiments of this disclosure.

[0034] For the sake of brevity, conventional techniques related to signal processing, data transmission, signal delivery, control, and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to illustrate exemplary functional relationships and / or physical connections between various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0035] refer to Figure 1 The diagram illustrates a functional block diagram of an autonomous vehicle 100 including an embodiment of an object detection system 110. The object detection system 110 is configured to receive images from a vehicle sensor system 128, detect one or more candidate objects in the images, and predict the object class of each detected object. The object detection system 110 is pre-trained to predict a predefined object class using predefined concepts associated with each of the predefined object classes. Examples of object classes include, but are not limited to, cars, buses, bicycles, motorcycles, and people.

[0036] Autonomous vehicle 100 typically includes a chassis 112, a body 114, front wheels 116, and rear wheels 118. The body 114 is mounted on the chassis 112 and substantially surrounds the components of the autonomous vehicle 100. The body 114 and chassis 112 may together form a frame. The front wheels 116 and rear wheels 118 are each rotatably connected to the chassis 112 near a corresponding corner of the body 114.

[0037] Autonomous vehicle 100, for example, is a vehicle automatically controlled to transport passengers from one location to another. While autonomous vehicle 100 is depicted as a passenger car in the illustrated embodiment, other examples of autonomous vehicles include, but are not limited to, motorcycles, trucks, sports utility vehicles (SUVs), recreational vehicles (RVs), boats, and aircraft. In one embodiment, autonomous vehicle 100 is a so-called Level 4 or Level 5 automation system. Level 4 system means “high automation,” referring to the performance of the automated driving system (ADS) in a specific driving mode across all aspects of a dynamic driving task, even if the human driver does not respond appropriately to intervention requests. Level 5 system means “full automation,” referring to the full-time performance of the ADS across all aspects of a dynamic driving task under all road and environmental conditions manageable by a human driver.

[0038] As shown in the figure, an autonomous vehicle 100 typically includes a propulsion system 120, a transmission system 122, a steering system 124, a braking system 126, a vehicle sensor system 128, an actuator system 130, at least one data storage device 132, at least one controller 134, and a vehicle communication system 136. In various embodiments, the propulsion system 120 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 122 is configured to transmit power from the propulsion system 120 to the front wheels 116 and the rear wheels 118 according to a selectable speed ratio. According to various embodiments, the transmission system 122 may include a stepped automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 126 is configured to provide braking torque to the front wheels 116 and the rear wheels 118. In various embodiments, the braking system 126 may include friction brakes, brake-by-wire brakes, regenerative braking systems such as electric motors, and / or other suitable braking systems. The steering system 124 influences the position of the front wheels 116 and the rear wheels 118. Although described as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 124 may not include a steering wheel.

[0039] The vehicle sensor system 128 includes one or more vehicle sensing devices 140a-140n that sense observable conditions of the external and / or internal environments of the autonomous vehicle 100. Examples of vehicle sensing devices 140a-140n include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The actuator system 130 includes one or more actuator devices 142a-142n that control one or more vehicle features, such as, but not limited to, propulsion system 120, transmission system 122, steering system 124, and braking system 126. In various embodiments, vehicle features may also include internal and / or external vehicle features, such as, but not limited to, doors, trunk, and cabin features, such as air, music, and lighting.

[0040] Vehicle communication system 136 is configured for wireless communication of information with other entities (“Vehicle-to-Everything (V2X) communication”). For example, vehicle communication system 136 is configured for wireless communication of information with other vehicles 148 (“Vehicle-to-Vehicle (V2V) communication”), driving system infrastructure (“Vehicle-to-Infrastructure (V2I) communication”), remote systems, and / or personal devices. In one embodiment, vehicle communication system 136 is a wireless communication system configured to communicate using the IEEE 802.11 standard or via a wireless local area network (WLAN) using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also considered within the scope of this disclosure. DSRC channels refer to one-way or two-way short-to-medium range wireless communication channels designed for automotive use, along with a corresponding set of protocols and standards.

[0041] Data storage device 132 stores data used for automatically controlling the autonomous vehicle 100. Data storage device 132 may be part of controller 134, separate from controller 134, or part of controller 134 and an independent system.

[0042] The controller 134 includes at least one processor 144 and a computer-readable storage device 146. The computer-readable storage device 146 may also be referred to as a computer-readable medium 146. In one embodiment, the computer-readable storage device 146 includes an embodiment of the object detection system 110. The processor 144 may be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 134, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any means generally used for executing instructions. The computer-readable storage device 146 may include, for example, volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and non-fail-to-recovery memory (KAM). KAM is a permanent or non-volatile memory that can be used to store various operational variables when the processor 144 is powered off. The computer-readable storage device 146 may be implemented using any of a variety of known memory devices, such as a programmable read-only memory (PROM), an electrical PROM (EPROM), an electrically erasable PROM (EEPROM), flash memory, or any other electrical, magnetic, optical, or combined memory device capable of storing data, some of which represents executable instructions used by the controller 134 in controlling the autonomous vehicle 100.

[0043] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by processor 144, the instructions receive and process signals from vehicle sensor system 128, execute logic, calculations, methods, and / or algorithms for automatically controlling components of autonomous vehicle 100, and generate control signals to actuator system 130 based on the logic, calculations, methods, and / or algorithms to automatically control one or more components of autonomous vehicle 100. Although in Figure 1 Only one controller 134 is shown in the figure. Alternative embodiments of the autonomous vehicle 100 may include any number of controllers 134 that communicate via any suitable communication medium or combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of the autonomous vehicle 100.

[0044] In various embodiments, one or more instructions of controller 134 are implemented to provide the ADS functionality as described with reference to one or more embodiments herein. Controller 134 or one of its functional modules is configured to implement the functionality described in one embodiment or a combination of embodiments of reference object detection system 110.

[0045] refer to Figure 2The diagram illustrates a functional block diagram of an embodiment of a feature extraction neural network training system 200 configured to train an embodiment of a feature extraction neural network 202. In one embodiment, the feature extraction neural network training system 200 is configured to pair predefined concepts associated with different object classes and image data associated with the object classes associated with the concepts using contrastive language-image pre-training (CLIP). The concepts appear in text form. The feature extraction neural network training system 200 includes a feature extraction neural network 202 to be trained, a similarity scoring module 204, and a contrastive loss module 206. The feature extraction neural network 202 includes a neural network text encoder 208 and a neural network image encoder 210.

[0046] The neural network text encoder 208 is configured to receive each of a predefined concept associated with a predefined object class and generate a concept embedding associated with the received concept. The concept embedding may be referred to as a low-level concept feature. The concept embedding is a 3D vector. Examples of object classes include, but are not limited to, cars, buses, bicycles, motorcycles, and people.

[0047] Examples of predefined concepts associated with passenger cars include, but are not limited to, passenger car windshield, passenger car fog lights, passenger car license plate lights, passenger car windshield wipers, passenger car headlights, and passenger car taillights. Examples of predefined concepts associated with buses include, but are not limited to, bus tires, bus doors, bus wheels, bus bumpers, bus streetlights, bus headlights, bus taillights, bus left-side indicator lights, bus right-side indicator lights, bus windows, bus headlights, bus windshields, bus roof windows, bus front license plates, bus headlights, and bus rearview mirrors.

[0048] Examples of predefined concepts associated with bicycles include, but are not limited to, bicycle seat, bicycle basket, bicycle lock, bicycle handlebars, bicycle bell, bicycle pedals, bicycle tire, bicycle spokes, bicycle wheel, and bicycle mirror. Examples of predefined concepts associated with motorcycles include, but are not limited to, motorcycle helmet, motorcycle chassis, motorcycle trim, motorcycle engine, motorcycle mirror, motorcycle handlebars, motorcycle spokes, and motorcycle bolts. Examples of predefined concepts associated with people include, but are not limited to, human head, human arm, human hand, human leg, human foot, human ear, human nose, human body, and human neck.

[0049] The neural network image encoder 210 is configured to receive images generated by searching for images based on object classes associated with predefined concepts provided to the neural network text encoder 208, and to generate image embeddings associated with the images. The images can be referred to as image data. The image embeddings can be referred to as low-level image features. The image embeddings can be 3D vectors.

[0050] The similarity scoring module 204 is configured to receive concept embeddings generated by the neural network text encoder 208 in response to receiving a concept as input, and image embeddings generated by the neural network image encoder 210 in response to an image obtained from a search. The similarity scoring module 204 is configured to generate a similarity score based on the received concept embeddings and image embeddings. The similarity scoring module 204 is configured to apply a cosine similarity function to the concept embeddings and image embeddings to generate the similarity score. The contrast loss module 206 is configured to use the similarity scores generated based on the concept embeddings associated with the concept and the image embeddings associated with each of the search-generated images to jointly train the neural network text encoder 208 and the neural network image encoder 210 using pairs of concepts and images. The feature extraction neural network training system 200 may include additional components to facilitate the training of the feature extraction neural network 202.

[0051] refer to Figure 3 The diagram illustrates an example of a method 300 representing an embodiment of training a feature extraction neural network 202. Method 300 is performed by a feature extraction neural network training system 200. Method 300 may be performed by hardware circuitry, firmware, software, and / or a combination thereof.

[0052] At 302, the concept associated with the object class is provided to the neural network text encoder 208. At 304, the neural network text encoder 208 generates a concept embedding associated with the received concept. At 306, a set of images is generated based on the execution of a search for images associated with the object class, and at 308, the images from the set of images are provided to the neural network image encoder 210. At 310, the neural network image encoder 210 generates an image embedding associated with the image. At 312, the similarity scoring module 204 receives the concept embedding and the image embedding, and at 314, the similarity scoring module 204 generates a similarity score based on the concept embedding and the image embedding. In one embodiment, the similarity scoring module 204 uses a cosine similarity function to generate the similarity score. At 316, the contrastive loss module trains the feature extraction neural network 202 based on image weighting according to the similarity score.

[0053] At 318, the feature extraction neural network training system 200 determines whether all images in a set of images generated based on the object class associated with the concept have been used to train the feature extraction neural network 202. If the feature extraction neural network training system 200 determines that not all images in a set of images generated based on the object class associated with the concept have been used to train the feature extraction neural network 202, then method 300 proceeds to 308, and the next image in the set of images is provided to the neural network image encoder 210.

[0054] If the feature extraction neural network training system 200 determines that all images in a set of images generated based on the object class associated with the concept have been used to train the feature extraction neural network 202, then at 320, the feature extraction neural network training system 200 determines whether all predefined concepts associated with the object class have been provided to the neural network text encoder 208.

[0055] If the feature extraction neural network training system 200 determines at 320 that not all predefined concepts associated with the object class have been provided to the neural network text encoder 208, then method 300 proceeds to 302, and the next concept associated with the object class is provided to the neural network text encoder 208. If the feature extraction neural network training system 200 determines that all predefined concepts associated with the object class have been provided to the neural network text encoder 208, then at 322, the feature extraction neural network training system 200 determines that the training of the feature extraction neural network 202 with respect to the object class is complete. Method 300 is repeated for each of the predefined object classes.

[0056] refer to Figure 4 This diagram illustrates a functional block diagram of an embodiment of an object detection system 110 including a feature extraction neural network 202. The object detection system 110 is communicatively coupled to a vehicle sensor system 128 of an autonomous vehicle 100. The object detection system 110 is configured to receive images from the vehicle sensor system 128, detect one or more objects in the images, and predict the object class of each detected object. The object detection system 110 is pre-trained to predict a predefined object class using predefined concepts associated with each of the predefined object classes. Examples of object classes include, but are not limited to, cars, buses, bicycles, motorcycles, and people.

[0057] The vehicle sensor system 128 includes one or more vehicle sensing devices 140a-140n that sense observable conditions of the external and / or internal environment of the autonomous vehicle 100 and generate sensing data. Examples of vehicle sensing devices 140a-140n include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The vehicle sensor system 128 is configured to generate images based on data sensed by the one or more vehicle sensing devices 140a-140n.

[0058] The object detection system 110 includes a feature extraction neural network 202, an object candidate module 400, a concept module 402, a confidence score module 404, and an object class prediction module 406. The feature extraction neural network 202 includes a neural network text encoder 208 and a neural network image encoder 210. The object detection system 110 may include additional components to facilitate the operation of the object classification system.

[0059] The object candidate module 400 is configured to receive an image generated by the vehicle sensor system 128. In one embodiment, the object candidate module 400 is configured to propose one or more candidate object regions in the image using a region-based Fast R-CNN algorithm. In another embodiment, the object candidate module 400 is configured to propose one or more candidate object regions in the image using the You Only Look Once (YOLO) method. In an alternative embodiment, other types of object detection methods may be used to propose one or more candidate object regions in the image. In one embodiment, the object candidate module 400 is configured to generate candidate object image regions for each possible object in the image. Each candidate object region is processed individually by the object detection system 110 to classify the detected objects in the candidate object region. The candidate object region may also be referred to as an image patch. Each image patch is at least a portion of the image and includes an object. The object may also be referred to as a candidate object.

[0060] The neural network image encoder 210 is configured to receive image patches including candidate objects and generate image embeddings associated with the image patches. Image embeddings can also be referred to as low-level features of the image patches. Image embeddings are 3D vectors.

[0061] Each object class is associated with a set of predefined concepts. As described above, the predefined concepts of each object class are used to pre-train the feature extraction neural network 202 of the object detection system 110. The predefined concepts have a text format. The concept module 402 is configured to individually provide each of the predefined concepts to the neural network text encoder 208. In one embodiment, the concept module 402 is configured to individually provide each of the predefined concepts associated with an object class, one object class at a time. Upon receiving a predefined concept for an object class, the neural network text encoder 208 is configured to generate a concept embedding associated with that predefined concept. The concept embedding may also be referred to as a low-level feature of the predefined concept or a low-level concept feature. The concept embedding is a 3D vector.

[0062] Examples of object classes include, but are not limited to, cars, buses, bicycles, motorcycles, and people. Examples of predefined concepts associated with cars include, but are not limited to, a car's windshield, car's fog lights, car's license plate lights, car's windshield wipers, car's headlights, and car's taillights. Examples of predefined concepts associated with buses include, but are not limited to, bus tires, bus doors, bus wheels, bus bumpers, bus streetlights, bus headlights, bus taillights, bus left-side indicator lights, bus right-side indicator lights, bus windows, bus headlights, bus windshields, bus roof windows, bus front license plates, bus headlights, and bus rearview mirrors.

[0063] Examples of predefined concepts associated with bicycles include, but are not limited to, bicycle seat, bicycle basket, bicycle lock, bicycle handlebars, bicycle bell, bicycle pedals, bicycle tire, bicycle spokes, bicycle wheel, and bicycle mirror. Examples of predefined concepts associated with motorcycles include, but are not limited to, motorcycle helmet, motorcycle chassis, motorcycle trim, motorcycle engine, motorcycle mirror, motorcycle handlebars, motorcycle spokes, and motorcycle bolts. Examples of predefined concepts associated with people include, but are not limited to, human head, human arm, human hand, human leg, human foot, human ear, human nose, human body, and human neck.

[0064] The confidence scoring module 404 receives image embeddings associated with image patches generated by the neural network image encoder 210 and concept embeddings associated with concepts of object classes generated by the neural network text encoder 208. The confidence scoring module 404 is configured to generate a confidence score based on the concept embeddings and image embeddings. A higher confidence score associated with a concept indicates a higher confidence that the concept exists in the image patch. A lower confidence score associated with a concept indicates a lower confidence that the concept exists in the image patch. The confidence score associated with the concept text is provided to the object class prediction module 406.

[0065] The concept associated with each object class is provided separately to the neural network text encoder 208. The concept embedding associated with each concept is provided to the confidence scoring module 404, and the confidence scoring module 404 provides the confidence score associated with the received concept to the object class prediction module 406. The object class prediction module 406 is configured to receive the confidence score associated with each of the concepts. Each of the concepts is associated with an object class.

[0066] In one embodiment, the object class prediction module 406 is configured to identify a predefined number of highest confidence scores associated with an image patch generated by the confidence scoring module 404. For example, the object class prediction module 406 may be configured to identify the five highest confidence scores generated by the confidence scoring module 404. In another example, the object class prediction module 406 is configured to identify the three highest confidence scores for conceptual text generated by the confidence scoring module 404.

[0067] The object class prediction module 406 is configured to identify the concept associated with each of the highest confidence scores identified. The object class prediction module 406 is configured to identify the object class associated with each of the identified concepts. The object class prediction module 406 is configured to determine the number of identified concepts associated with each of the object classes and select the object class associated with the largest number of identified concepts as the predicted object class in the image patch. The image detection system 110 repeats this process for each image patch associated with the image received from the vehicle sensor system 128.

[0068] The ADS is configured to receive one or more predicted object classes present in an image generated by the vehicle sensor system 128, and to manage the performance of the autonomous vehicle 100 based on the presence of a predicted object class in the vicinity of the autonomous vehicle 100.

[0069] refer to Figure 5The diagram illustrates an example of a method 500 for classifying objects in an image using an embodiment of an object detection system 110 at an autonomous driving system (ADS). Method 500 is performed by a controller 134 including the object detection system 110. Method 500 may be performed by the controller 134 in conjunction with other components of the autonomous vehicle 100. Method 500 may be performed by hardware circuitry, firmware, software, and / or a combination thereof.

[0070] At 502, the object candidate module 400 receives an image generated by the vehicle sensor system 128. At 504, the object candidate module 400 detects one or more objects in the image and generates an image patch for each detected object. At 506, the neural network image encoder 210 receives the image patch and generates an image embedding associated with the image patch.

[0071] At 508, the concept module 402 provides the neural network text encoder 208 with one of a plurality of predefined concepts. At 510, the neural network text encoder 208 generates concept embeddings associated with the concept. Each of the concepts is predefined and associated with an object class. The neural network text encoder 208 is configured to generate concept embeddings associated with the concepts based on pre-training of the feature extraction neural network 202, which is based on pairing concepts with image data associated with the object classes associated with the concepts.

[0072] At 512, the confidence scoring module 404 generates a confidence score based on the image embedding associated with the image patch generated by the neural network image encoder 210 and the concept embedding associated with the concept generated by the neural network text encoder 208. In one embodiment, the confidence scoring module 404 applies a cosine similarity function to the image embedding and the concept embedding to generate the confidence score. At 514, the confidence scoring module 404 sends the confidence score associated with the concept to the object class prediction module 406.

[0073] At 516, the object detection system 110 determines whether multiple predefined concepts have been processed for the image patch to generate a confidence score for each of the concepts. If the object detection system 110 determines that multiple predefined concepts have not been processed for the image patch to generate a confidence score for each of the concepts, then method 500 proceeds to 508, where the concept module 402 provides another of the multiple predefined concepts to the neural network text encoder 208.

[0074] If the object detection system 110 determines that multiple predefined concepts have been processed for an image patch to generate a confidence score for each of the concepts, at 518, the object class prediction module 406 identifies the highest confidence score among the predefined number of predefined values ​​related to the image patch received from the confidence score module 404. At 520, the object class prediction module 406 identifies the concept associated with each of the identified confidence scores. The object class prediction unit 406 identifies the object class associated with the majority of the identified concepts and generates the identified object class as the predicted object class of the object in the image patch. The ADS manages the performance of the autonomous vehicle 100 based on the presence of the predicted object class in the image generated by the vehicle sensor system 128.

[0075] At 524, the object detection system 110 determines whether all image blocks associated with the image generated by the vehicle sensor system 128 have been processed. If the object detection system 110 determines that all image blocks associated with the image generated by the vehicle sensor system 128 have not been processed, then method 500 proceeds to 506, where the neural network image encoder 210 receives another image block associated with that image and generates an image embedding associated with that image block for processing by the object detection system 110.

[0076] If the object detection system 110 determines that all image blocks related to the images generated by the vehicle sensor system 128 have been processed, then method 500 returns to 502, where a new image is received from the vehicle sensor system 128 for processing by the object detection system 110.

[0077] refer to Figure 6A and Figure 6B The table shows examples of confidence scores associated with example concepts of different object classes, including cars, buses, bicycles, motorcycles, and people, generated by the confidence scoring module 404 of an embodiment of the object detection system 110. In this example, the object class prediction module 406 selects the five highest confidence scores associated with image patches generated by the confidence scoring module 404. In this example, the five highest confidence scores generated by the confidence scoring module 404 are 0.30, 0.29, 0.28, 0.27, and 0.26.

[0078] The object class prediction module 406 identifies the concept associated with each of the identified confidence scores. The identified concepts and their associated confidence scores are... Figure 6AThe concepts shown in the table are in bold. The concept associated with a confidence score of 0.30 is the bus's front license plate. The concepts associated with a confidence score of 0.29 are the bus's doors, bumper, and right-side indicator light. The concepts associated with a confidence score of 0.28 are the bus's streetlights, left-side indicator light, and rearview mirror. The concepts associated with a confidence score of 0.27 are the bus's tires, headlights, windows, and headlights. The concepts associated with a confidence score of 0.26 are the bus's taillights, wheels, headlights, and windshield. In this example, the object class prediction module 406 identifies the bus as the object class associated with the majority of the identified concepts and generates a predicted object class for the bus as an object in the image patch.

[0079] refer to Figure 7 The diagram illustrates an example of a method 700 for classifying objects in an image using an embodiment of an object detection system 110 at an autonomous driving system (ADS). Method 700 is performed by a controller 134, which includes one embodiment of the object detection system 110. Method 700 may be performed by the controller 134 in conjunction with other components of the autonomous vehicle 100. Method 700 may be performed by hardware circuitry, firmware, software, and / or a combination thereof.

[0080] At 702, an image embedding associated with at least a portion of an image generated by the vehicle sensor system 128 is generated at the neural network image encoder 210. At least a portion of the image includes objects. At 704, a concept embedding associated with each of a plurality of concepts is generated at the neural network text encoder 208. Each of the plurality of concepts is associated with one of at least two object classes. At 706, a confidence score for each of the plurality of concepts is generated at the confidence scoring module 404. The confidence score for each of the plurality of concepts is generated based on the image embedding and the concept embedding associated with the concepts. At 708, at the object class prediction module 406, a predicted object class for the object is generated based on the association between a concept set of the plurality of concepts having at least two of the highest values ​​of the generated confidence scores and one of at least two object classes associated with the majority of the concept set. At 710, the performance of the autonomous vehicle, partially based on the predicted object class of the object, is managed at the controller 134.

[0081] Classifying objects in images captured by the vehicle sensor system 128 of autonomous vehicle 100 in low-visibility environments can be challenging due to the lack of clarity in representative features associated with objects that are typically relatively easy to identify in images captured under normal weather conditions. Classifying objects in images captured under low-visibility conditions can be achieved by using an object detection system 110, which includes a feature extraction neural network 202, to identify concepts associated with object classes and to classify detected objects in the image using the identified concepts. Using concepts to classify objects provides a higher level of abstraction of the detected objects compared to classifying objects using representative features.

[0082] While at least one exemplary embodiment has been given in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. An autonomous driving system (ADS) for an autonomous vehicle, comprising an object detection system, the object detection system comprising: A neural network image encoder is configured to generate an image embedding associated with at least a portion of an image generated by a vehicle sensor system, the at least portion of which includes an object; A neural network text encoder is configured to generate concept embeddings associated with each of a plurality of concepts, each of which is a component of an object class, and each of the plurality of concepts is associated with a dependency of at least two object classes. The confidence scoring module is configured to generate a confidence score for each of the plurality of concepts based on the image embedding and the concept embedding associated with the concept; An object class prediction module is configured to generate a predicted object class for the object based on an association between a concept set of the plurality of concepts having at least two of the highest values ​​of the generated confidence scores and one of the at least two object classes associated with the majority of the concept set; as well as The processor is configured to manage the performance of the autonomous vehicle in part based on the prediction object class of the object.

2. The system according to claim 1, wherein, Each of the plurality of concepts is associated with one of the at least two object classes that include the predefined concepts.

3. The system according to claim 1, wherein, The object classes among the at least two object classes include at least two of the following: cars, buses, bicycles, motorcycles, and pedestrians.

4. The system of claim 1 further includes an object candidate module configured to detect the object in the image received from the vehicle sensor system and to generate at least a portion of the image based on the position of the detected object in the image.

5. The system according to claim 4, wherein, The object candidate module is configured to implement either the region-based Fast R-CNN algorithm or the YOLO method (You Only Look Once).

6. The system according to claim 1 further includes a feature extraction neural network, the feature extraction neural network comprising the neural network image encoder and the neural network text encoder, wherein, The feature extraction neural network is configured to be pre-trained to connect the plurality of concepts and image data associated with each of the at least two object classes through a feature extraction neural network training system.

7. The system according to claim 1, wherein, The confidence scoring module is configured to generate a confidence score for each of the plurality of concepts by applying a cosine similarity function to the image embedding and the concept embedding associated with the concept.

8. A computer-readable medium comprising instructions stored thereon, the instructions being configured to classify objects detected in an image at an automated driving system (ADS) of an autonomous vehicle, wherein, when executed by a processor, the instructions cause the processor to: Generate an image embedding associated with at least a portion of an image generated by a vehicle sensor system, wherein at least a portion of the image includes an object; Generate concept embeddings associated with each of a plurality of concepts, each of which is a textual description of a component of an object class, and each of which is associated with a dependency of at least two object classes; Based on the image embedding and the concept embedding associated with the concept, a confidence score is generated for each of the plurality of concepts; Based on the association between a concept set of the plurality of concepts having at least two of the highest generated confidence scores and one of the at least two object classes associated with the majority of the concept set, a predicted object class for the object is generated; and The performance of the autonomous vehicle is managed in part based on the predictive object class of the object.

9. The computer-readable medium of claim 8, further comprising instructions for causing the processor to generate a concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes including a predefined concept.

10. The computer-readable medium of claim 8, further comprising instructions for causing the processor to generate a concept embedding associated with each of the plurality of concepts, each of the plurality of concepts being associated with one of the at least two object classes including at least two of cars, buses, bicycles, motorcycles, and pedestrians.

Images