System and method for annotating images to train machine learning models

Abstract

The invention relates to a system and method for training a machine learning model for automatic driving. The system includes a plurality of vehicles configured to capture at least a surrounding view in front of the vehicle, a machine learning training system, and an authentication computing device. A machine learning training system is configured to receive a captured image from a vehicle. The verification computing device is configured to verify whether the machine learning model correctly identifies a light indicator of the vehicle shown in the captured image. The verification device may determine an inconsistency between a predicted light indicator and a correct light indicator of the vehicle. Upon determining that the at least one vehicle has an inconsistency, the verification computing device is configured to modify the light indicator indicia and the revised indicia. The modified annotation may then be sent to the machine learning model, and used to train the machine learning model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 344,303, filed on May 20, 2022, entitled “Systems and Methods for Labeling Images for Training Machine Learning Models,” the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] Embodiments of the present disclosure relate to systems and methods for annotating images to train machine learning models. More specifically, embodiments of the present disclosure relate to systems and methods for training machine learning models to detect light indicators on one or more vehicles as part of an autonomous driving system. Background Art

[0004] Automated driving systems (e.g., self-driving systems) typically acquire images of the road and surrounding vehicles and input these images into a trained machine learning model to control the vehicle with no or limited user input. The machine learning models used in such systems are typically trained by first capturing millions or billions of images and then annotating these images with feature labels (indicating features to be identified in the vehicle's surroundings). For example, these features may include curbs, painted lines, other vehicles, road cones, traffic lights, and other objects found on the road. Once the machine learning model is trained to recognize these features, the machine learning model can be downloaded and stored in the vehicle's memory, enabling the vehicle to operate in an automated or semi-automated driving mode. Summary of the invention

[0005] The innovations described in the claims each have multiple aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of the claims, some of the main features of the disclosure will now be briefly described.

[0006] One aspect of the present disclosure includes a system for annotating an image to train a machine learning model to detect light indicators on a vehicle. The system includes: acquiring an image of one or more vehicles on a road; identifying a location of each of the one or more vehicles; displaying a graphical indicator on each of the one or more vehicles to indicate that the vehicle is detected by the system; and receiving an indication of whether the light indicator is activated or inactivated on each of the one or more vehicles to annotate the image for the machine learning model.

[0007] In the system, acquiring images may include acquiring images from a plurality of vehicles having an autonomous driving system, and acquiring images may further include acquiring images of the plurality of vehicles when the autonomous driving system determines that the light indicator detection is incorrectly determined by the autonomous driving system.

[0008] In the system, identifying the location of each of the one or more vehicles may include identifying the vehicle in the image and determining graphical coordinates of the vehicle in the image.

[0009] In the system, displaying the graphical indicator on each of the one or more vehicles may include displaying a bounding box around each of the one or more vehicles in the acquired image.

[0010] In the system, identifying the location of each of the one or more vehicles may include performing image segmentation on the acquired images, and the image segmentation generates a region of each acquired image that can correspond to a vehicle.

[0011] In the system, receiving an indication of whether the light indicator is activated or inactivated may include receiving a mouse selection from a user that labels the vehicle as having an activated light indicator or an inactivated light indicator.

[0012] In the system, receiving an indication of whether the light indicator is activated or inactivated may include receiving an indication of whether a brake light is activated or inactivated.

[0013] In the system, receiving an indication of whether the light indicator is activated or inactivated may include receiving an indication of whether the turn signal is activated or inactivated.

[0014] Another aspect of the present disclosure includes a system for annotating images to train a machine learning model to detect light indicators on vehicles. The system includes: acquiring an image of one or more vehicles on a road; identifying a location of each of the one or more vehicles in the acquired image; determining whether the light indicator is indicated as activated or inactivated by an autonomous driving system in each of the one or more vehicles; determining one or more vehicles based on the image of the one or more vehicles that have a false prediction of whether the light indicator is activated or inactivated; and annotating the image with the false prediction with a correct indication of whether the light indicator is activated or inactivated.

[0015] In the system, identifying the location of each of the one or more vehicles may include identifying the vehicle in the image and determining graphical coordinates of the vehicle in the image.

[0016] In the system, acquiring images may include acquiring images from a plurality of vehicles having an autonomous driving system. Acquiring images may also include acquiring images from the plurality of vehicles when the autonomous driving system determines that the light indicator detection is incorrectly determined by the autonomous driving system.

[0017] In the system, a graphical indicator is displayed on each of the one or more vehicles to indicate that the vehicle is detected by the system. Displaying the graphical indicator on each of the one or more vehicles may also include: displaying a bounding box around each of the one or more vehicles in the acquired image.

[0018] In the system, an indication as to whether a light indicator of each of one or more vehicles is activated or inactivated may be predicted by an autonomous driving system of each of the vehicles.

[0019] In this system, a false prediction may represent an inconsistency between the position of the light indicator and the vehicle.

[0020] The system may also include receiving an updated light indicator, receiving a mouse selection from a user to annotate the vehicle with the light indicator based on the location of the vehicle.

[0021] In the system, the indication as to whether the light indicator is activated or inactivated may be an indication as to whether the brake light is activated or inactivated.

[0022] In the system, the indication as to whether the light indicator is active or inactive may be receiving an indication as to whether the turn signal is active or inactive.

[0023] Another aspect of the present disclosure includes a method for annotating images to train a machine learning model to detect light indicators on vehicles. The method includes: acquiring images of one or more vehicles on a road; identifying a location of each of the one or more vehicles; annotating an indication of whether a light indicator is activated or inactivated on each of the one or more vehicles; determining one or more vehicles with false predictions from the images of the one or more vehicles; and receiving an updated indication of whether a light indicator is activated or inactivated on the vehicles with the false predictions.

[0024] In the method, identifying the location of each of the one or more vehicles may include identifying the vehicle in the image and determining graphical coordinates of the vehicle in the image.

[0025] In the method, acquiring images may include acquiring images from a plurality of vehicles having autonomous driving systems.

[0026] In the method, acquiring images may include acquiring images from one or more vehicles when light indicator detection is incorrectly determined by the autonomous driving system of each vehicle.

[0027] For purposes of summarizing the present disclosure, certain aspects, advantages, and novel features of the innovations have been described herein. It is to be understood that not all of these advantages may be realized according to any particular embodiment. Thus, the innovations may be implemented or performed in a manner that achieves or optimizes one advantage or group of advantages as taught herein without achieving other advantages as also taught or suggested herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Embodiments of the present disclosure will now be described by way of non-limiting examples and with reference to the accompanying drawings.

[0029] Figure 1 is a block diagram illustrating communications between multiple vehicles, a network server, and a verification computing device.

[0030] Figure 2 is a schematic diagram illustrating an example of a vehicle.

[0031] Figure 3A It depicts Figure 2 A block diagram of an embodiment of the architecture of a vehicle autonomous driving system in FIG.

[0032] Figure 3B It depicts Figure 1 A block diagram of an embodiment of the architecture of a machine learning training system in FIG.

[0033] Figure 4 An example of a vehicle having a camera for capturing images of other vehicles is illustrated.

[0034] Figure 5 An example of a captured image is illustrated, including identified light indicators associated with each vehicle in the image.

[0035] Fig. 6A An example of identified light indicator verification of a vehicle in a captured image is illustrated.

[0036] Figure 6B An example of an interactive graphical user interface for annotating a graphical indicator of a vehicle in a captured image is illustrated.

[0037] Figure 7 An example of labeling a graphical indicator with the correct light indicator of a vehicle in a captured image is illustrated.

[0038] Figure 8is a flow diagram illustrating one embodiment of an example process that a verification computing device may perform to detect one or more vehicles in a captured image that have inconsistencies between identified light indicators and actual light indicators of the vehicles.

[0039] Fig. 9 is a flow diagram illustrating one embodiment of an example process that a validation computing device may perform to train a machine learning model.

[0040] FIG. 10A to FIG. 10D Examples of light indicator detection in various environments are illustrated.

[0041] Fig.11 Illustrated are example interactive user interfaces that a user may use. DETAILED DESCRIPTION

[0042] Although certain preferred embodiments and examples are disclosed below, the subject matter of the present invention is not limited to the specific embodiments disclosed, but also includes other alternative embodiments and / or uses and their modifications and equivalents. Therefore, the scope of the claims attached hereto is not limited to any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process can be performed in any suitable order and are not necessarily limited to any specific disclosed order. Various operations can be described as multiple discrete operations, which can help understand certain embodiments in this way; however, the order of description should not be interpreted as implying that these operations are sequentially dependent. In addition, the structures, systems and / or devices described herein can be implemented as integrated components or as separate components. For the purpose of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not all of these aspects or advantages are achieved by any particular embodiment. Therefore, for example, various embodiments can be carried out in a way that realizes or optimizes an advantage or a group of advantages as taught herein, without realizing other aspects or advantages as also taught or implied herein.

[0043] One or more aspects of the present application correspond to systems and methods for training machine learning models associated with autonomous driving systems. An exemplary machine learning model can be used to detect nearby vehicles and determine whether these nearby vehicles have detectable light indicators or signals. In some embodiments, the light signals can be brake lights, turn indicators, headlights, or any other lighting indicators on the vehicle. The light signals can also be brake lights, turn indicators, etc. associated with a trailer connected to the vehicle. In some embodiments, detectable light indicators can be located on the road, such as traffic lights, flashing stop signs, or other lighting signals on a typical road. Based on the determined light indicators, the autonomous driving system can predict the travel path, speed, etc. of nearby detected vehicles.

[0044] Embodiments of the disclosed technology correspond to systems and methods for training machine learning models by more accurately annotating light indicators in images captured from vehicles or road features (e.g., images obtained from an image sensor of a camera mounted on a vehicle). More specifically, these systems and methods are used to obtain images captured by a camera mounted on a vehicle while the vehicle is traveling on a road. These captured images can then be uploaded to a server or external system so that the images can be annotated with various features. The uploaded images can be displayed to a user (e.g., a human user, a software agent) so that the user can identify and annotate the state of the light indicators found within the captured image for use in training data. For example, the captured image can be a vehicle with a left turn signal illuminated. The user can select the left turn signal illuminated through a user interface, and then store the label with the picture for use in training an autonomous driving machine learning model or a semi-autonomous driving machine learning model, such as a visual model. As another example, the image can be a traffic light, and the user can annotate the picture as showing a red traffic light on. The terms "image" and "video clip" in this disclosure can be used interchangeably, and these terms have similar meanings. For example, if a set of continuously captured images is 300, a 10 second video clip can be played at a rate of 30 frames per second. Therefore, the 300 captured images can have the same meaning as the 10 second video clip. The number of images and the video clip rate are presented only as examples, and different numbers of images and rates can be used depending on the specific application.

[0045] In some embodiments, the annotation system used by the user to annotate the image may include certain elements to improve the accuracy of the annotation. For example, the system may automatically outline each vehicle in the image, such as a bounding box, so that the user can select a specific vehicle to annotate. The user can select a bounding box around the vehicle (e.g., through an interactive user interface), and then display various options for annotating light indicators on the vehicle. These options may include a left turn signal, a right turn signal, a brake light, or other similar features of the vehicle. This allows the user to annotate different features to multiple vehicles in a single captured image to improve the accuracy of the annotation process and enhance the ability of the image to be used to train a machine learning model to identify the light indicators of vehicles on the road.

[0046] In some embodiments, a vehicle capturing and uploading images may upload only those images where light indicator prediction errors are found. For example, a vehicle may be running autonomous driving software and identify in a captured image that the vehicle ahead has no brake lights on. However, the vehicle may also detect that the vehicle ahead is slowing down due to traffic. In this case, the brake lights should probably have been on, so captured images identified as not having brake lights on may be uploaded to a server for manual annotation of brake lights to improve future models for autonomous driving.

[0047] In some embodiments, the vehicle uploading the image may be running autonomous driving software in a stealth mode, where the vehicle is not driving in autonomous mode, but the vehicle is still capturing images and determining actions for the vehicle as if the system were controlling the vehicle. In this stealth mode, the vehicle can identify potential errors in how it handles the light indicators and upload the images that led to the potential errors to the server for processing, review, and user update annotation.

[0048] In order to resolve errors of the autonomous driving system related to light indicator determination, the machine learning model can be trained by the above method by updating the machine model with correct data. Specifically, incorrect light indicator data (e.g., an image or video clip) based on the machine learning model can be corrected by receiving correct light indicator data. For example, the correct light indicator data can be superimposed on the incorrect light indicator data, and the machine learning model can be trained using the superimposed data. Training can include updating or modifying multiple parameters and attributes related to the machine learning model.

[0049] Various aspects of machine learning model training will be described in certain examples and embodiments, which are for illustration only. Although the examples and embodiments described herein will focus on specific calculations and algorithms for illustrative purposes, those skilled in the art will understand that these examples are illustrative only and are not intended to be limiting.

[0050] Figure 1 is a block diagram illustrating an embodiment of a system 100. The system 100 may include a network that connects a plurality of vehicles 110, a machine learning training system 120, and a verification computing device 130. Exemplarily, various aspects associated with the machine learning training system 120 may be implemented as one or more components associated with one or more functions or services. These components may correspond to software modules implemented or executed by one or more external computing devices, which may be separate independent external computing devices. Therefore, the components of the machine learning training system 120 should be viewed as logical representations of the service without requiring any specific implementation on one or more external computing devices.

[0051] like Figure 1 As shown, the network 160 connects the vehicle 110 and the verification computing device 130 to the machine learning training system 120. For example, the network 160 may include any combination of wired and / or wireless networks, such as one or more direct communication channels, local area networks, wide area networks, personal area networks, and / or the Internet. In some embodiments, the network 160 may include one or more wireless networks, such as a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a long-term evolution (LTE) network, 5G communication, or any other type of wireless network. The network 160 may communicate through the Internet or any other type of network using protocols and components. For example, the protocols used by the network 160 may include hypertext transfer protocol (HTTP), HTTP security (HTTPS), message queue telemetry transmission (MQTT), constraint application protocol (CoAP), etc. The protocols and components for communicating through the Internet or any other type of communication network are known to those skilled in the art, so they are not described in detail herein. In some embodiments, wireless communications via the network 160 may be performed on one or more secure networks, such as communicating via SSL (e.g., 256-bit, military-grade encryption) using encrypted data. The various communication protocols discussed herein are only examples, and the present application is not limited thereto.

[0052] Figure 1The vehicles 110 in the system can be connected to the machine learning training system 120. The vehicles 110 can be a group of multiple vehicles. In some embodiments, each of the vehicles 110 is configured to capture images of its surroundings, including nearby vehicles, traffic signals, surrounding environment, etc. The captured images can be encoded into a video file based on the resolution specification of each camera in the camera and transmitted (e.g., uploaded) to the machine learning training system 120 via the network 160. In some embodiments, each vehicle 110 may include one or more microprocessors and circuit devices configured to establish a wireless communication channel to connect to the network 160. In order to establish a wireless communication channel, each of the vehicles 110 can periodically (or continuously) scan and detect any nearby wireless signals. In another embodiment, the operator of the vehicle 110 can manually establish a wireless connection and connect to the network 160. For example, the operator can access a nearby Wi-Fi router to wirelessly connect the vehicle 110 to the network 160.

[0053] Figure 1 The machine learning training system 120 in the embodiment can train a machine learning model 124, and the model can be provided to the vehicle 110 for use in autonomous driving or semi-autonomous driving. Exemplarily, the machine learning training system 120 may include a machine learning model 124, a routing component 126, and a network server 128. The network server 128 is configured to store captured images received from the vehicle 110.

[0054] like Figure 1 As shown, the machine learning model 124 can be part of the machine learning training system 120. In some embodiments, the machine learning model 124 is included in the machine learning training system 120. In other embodiments, the machine learning model 124 is an independent component and is interconnected with other components in the machine learning training system, such as a network server 128.

[0055] In some embodiments, the machine learning model 124 is configured to identify features in the captured images stored in the network server 128. For example, the features may include curbs, painted lines, other vehicles, road cones, traffic signals, and other objects found on the road. Thus, the machine learning model 124 may be or include a vision-only model, such as a convolutional neural network, a transformer network, a fully connected network, a combination thereof, and the like.

[0056] Among these features, the machine learning model 124 may be configured to identify light indicators of surrounding vehicles located in front of the vehicle 110 (or surrounding vehicles captured by a front camera of the vehicle 110). The identified light indicators may be displayed on the vehicles contained in the image.

[0057] exist Figure 1In the example of , the verification computing device 130 is connected to the machine learning training system 120 via the network 160. In some embodiments, one or more authorized analysts (including managers, developers, supervisors, administrators, etc.) can use the verification computing device 130 to access the network server 128. The verification computing device 130 can be any computing device, such as a desktop computer, a laptop computer, a personal computer, a tablet computer, a wearable computer, a server, a personal digital assistant (PDA), a hybrid PDA / mobile phone, a mobile phone, a smart phone, a set-top box, a voice command device, a digital media player, etc. The verification computing device 130 can execute an application (e.g., a browser, a stand-alone application, etc.) that allows a user to access an interactive user interface, view images, analyze, summarize data, and / or other content described herein. In addition, the verification computing device 130 can have a display and an input device, through which the user can interact with the user interface components.

[0058] The verification computing device 130 may be configured to access the network server 128 via the network 160 and download one or more images or video clips stored in the network server 128. In some embodiments, the verification computing device 130 is configured to identify (or predict) the light indicators of the vehicle in the downloaded image or video clip. When identifying the light indicators of the vehicle, the verification computing device 130 may be configured to use one or more attributes or algorithms stored in the machine learning model 124. For example, an analyst may download a captured image from the network server 128 and execute instructions to the machine learning model to identify the light indicators on the vehicle contained in the downloaded image.

[0059] In some embodiments, the verification computing device 130 is configured to determine whether the machine learning model correctly identified features in the captured image. In these embodiments, the analyst can use the verification computing device 130 to determine whether the machine learning model 124 correctly identified the light indicators of surrounding vehicles in the captured image. For example, the analyst can analyze the image or video clip to determine whether there are inconsistencies between the vehicle light indicators identified in the captured image and the actual light indicators, and analyze the vehicle's travel path.

[0060] In some embodiments, the verification computing device 130 can be configured to flag the images after determining that the light indicators of one or more vehicles in the images are incorrectly determined. The analyst can correct the flagged images. In some embodiments, the corrected images can be uploaded to the network server 128. In some embodiments, the analyst can use the corrected images as training data to train the machine learning model 124. For example, the training data can be fed into the machine learning model 124. In this example, the machine learning model can update or modify its algorithms or properties associated with the trained machine learning model. The trained machine learning model can be provided to the vehicle 110 via the routing component 126. The vehicle 110 can therefore execute the model based on the image input, such as by computing a forward propagation.

[0061] Figure 2 is a schematic diagram illustrating an example of a vehicle 110 . Figure 2 A top view of a vehicle 110 is shown, illustrating an arrangement of multiple image sensors or cameras 220, 230, 240 (e.g., cameras configured for mounting at locations inside the vehicle or at locations outside the vehicle). In some embodiments, the vehicle 110 is configured to capture images of the surroundings. In some embodiments, the vehicle 110 has an autonomous driving function (e.g., self-driving). In some embodiments, the cameras are placed at different locations inside and outside the vehicle 110. For example, in Figure 2 In the embodiment, the front camera 220 is installed on the front side of the vehicle 110, such as on the upper side of the front windshield. The pillar camera 230 is installed on both sides of the vehicle 110, such as on the pillars of the vehicle 110. For example, the pillar camera 230 can be installed inside the pillar. The relay camera 240 is installed at the relay position on both sides of the vehicle 110.

[0062] In some embodiments, the cameras 220, 230, 240 capture images of the road and vehicles around the vehicle 110. In these embodiments, the front camera 220 captures images in front of the vehicle 110. The pillar camera 230 is configured to capture images on both sides of the vehicle 110. The relay camera 240 is configured to capture images behind the vehicle 110.

[0063] In some embodiments, the vehicle 110 includes at least one controller having one or more microprocessors and circuit devices configured to establish a wireless communication channel connected to the network 160. The controller can transmit (e.g., feed or upload) the captured image to the network server 128 via the network 160. The captured image can also be encoded into a video file based on the resolution specification of each camera in the camera and transmitted to the network server 128.

[0064] In some embodiments, the vehicle 110 includes a vehicle autonomous driving system 210. The vehicle autonomous driving system 210 can control the vehicle 110 to perform autonomous driving (e.g., self-driving). The autonomous driving system 210 can access the captured image and identify the surrounding features based on the machine learning model provided by the machine learning training system 120. For example, these features can include the light indicator of each surrounding vehicle, which is displayed on the image captured by the front camera 220. Features can also include road surface information, such as curbs, painted lines, traffic cones, traffic lights, and other objects on the road. The communication configuration between the cameras 220, 230, 240 and the autonomous driving system 210 can be direct communication or indirect communication through a wired connection using a communication cable or bus. Various wired communication networks can be used, such as a controller area network (CAN), and network protocols can be specified based on specific applications.

[0065] Figure 3A is a block diagram depicting one embodiment of the architecture of the autonomous driving system 210. The overall architecture of the autonomous driving system 210 includes an arrangement of computer hardware and software components that can be used to implement embodiments of the present disclosure. As shown, the autonomous driving system 210 includes a processing unit 302, an input / output device interface 304, a computer readable medium 306, and a network interface 308, all of which can communicate with each other by way of a communication bus. The components of the autonomous driving system 210 can be physical hardware components installed within the vehicle 110.

[0066] The input / output device interface 304 may provide a connection with the cameras 220, 230, 240. Therefore, the input / output device interface 304 may receive captured images or video files from the cameras 220, 230, 240. The received images or video files may be stored in the computer-readable medium 306. The computer-readable medium 306 may be an internal or external drive and may be in communication with the memory 310.

[0067] The memory 310 may include computer program instructions that the processing unit 302 executes to implement one or more embodiments. The memory 310 typically includes RAM, ROM, or other persistent or non-transient memory. The memory 310 may store an operating system 314 that provides computer program instructions for use by the processing unit 302 in the overall management and operation of the autonomous driving system 210. The memory 310 may also include computer program instructions and other information for implementing various aspects of the present disclosure. For example, the memory 310 includes a detection vehicle input component 316 that is configured to obtain captured images or video files from the cameras 220, 230, 240. The memory 310 also includes an autonomous driving model 318 that is configured to provide vehicle autonomous driving functions by identifying surrounding features of the vehicle 110. For example, features may include curbs, painted lines, other vehicles, traffic cones, traffic lights, and other objects on the road. In some embodiments, the machine learning model 124 may be fed into the autonomous driving model 124 through the network 160 so that the autonomous driving model 318 uses the attributes, parameters, and algorithms implemented in the machine learning model 124. In some embodiments, the autonomous driving model 318 may be updated using the trained machine learning model.

[0068] In some embodiments, the processing unit 302 may also communicate with the memory 310 and further provide output information for automatic vehicle driving through the input / output device interface 304. Exemplarily, the processing unit 302 may receive a light indication for each vehicle identified by the autonomous driving model 318. In response to receiving the identified vehicle light indication, the processing unit 302 may execute one or more commands to the autonomous driving system 210 to adjust its autonomous driving based on the light indication. For example, after obtaining the detected vehicle from the detection vehicle input component 316, the autonomous driving model 318 determines, based on a plurality of machine learning attributes, that one of the detected vehicles has turned on a right turn signal and has identified a right turn signal indication. In this example, the processing unit 302 may execute a command to the autonomous system to reduce the speed of the vehicle 110 or turn the vehicle 110 in a particular direction.

[0069] Network interface 308 may provide connectivity to one or more networks or computing systems, such as Figure 1 In some embodiments, the processing unit 302 executes sending data to the network server 128 or receiving data from the network server 128 through the network interface 308.

[0070] Figure 3B Depicted is a verification computing device 130 (eg Figure 1130). The overall architecture of the verification computing device 130 includes an arrangement of computer hardware and software components that can be used to implement embodiments of the present disclosure. As shown, the verification computing device 130 includes a processing unit 322, an input / output device interface 324, a computer readable medium 326, and a network interface 328, all of which can communicate with each other via a communication bus. One or more authorized analysts (including managers, developers, supervisors, administrators, etc.) can use the verification computing device 130 to execute instructions related to one or more embodiments of the present disclosure.

[0071] The input / output device interface 324 may provide a connection to the network server 128. Thus, the processing unit 322 may access the network server 128 to transmit or receive data through the input / output device interface 324. In some embodiments, the data received from the network server 128 is stored in the computer-readable medium 326. The computer-readable medium 326 may be an internal drive or an external drive and may communicate with the memory 310.

[0072] The memory 330 may include computer program instructions that the processing unit 322 executes to implement one or more embodiments. The memory 330 typically includes RAM, ROM, or other persistent or non-transient memory. The memory 330 may store an operating system 334 that provides computer program instructions for use by the processing unit 332 in the overall management and training of the machine learning model 124. The memory 330 may also include computer program instructions and other information for implementing various aspects of the present disclosure. For example, the memory 330 includes an input processing component 336, a graphic indicator overlay component 338, a light indicator display component 340, a machine learning model validation component 342, and a machine learning model training component 344.

[0073] Figure 3BThe input processing component 336 in is configured to obtain captured images around the vehicle 110. The input processing component 336 is configured to access captured images of surrounding vehicles, where the images are stored in the network server 128. The surrounding images of the vehicle 110 can be captured using a camera installed on each vehicle 110 and transmitted to the network server 128. The captured images can be used for autonomous driving (e.g., self-driving) by the machine learning model 124 to identify one or more features in the vehicle's surrounding environment. For example, features may include curbs, painted lines, other vehicles, traffic cones, traffic lights, and other objects on the road. In some embodiments, captured images are used to train the machine learning model 124. For example, in response to determining that the machine learning model incorrectly identifies one of the features in the image, the analyst can use the verification computing device 130 to annotate the image to correct the identified features, and use the annotated images (annotated images) to train the machine learning model.

[0074] The graphic indicator overlay component 338 can be configured to generate a graphic indicator for each vehicle in the captured image. For example, the graphic indicator can be included or presented in an interactive user interface that presents an image captured by the vehicle. The graphic indicator can be box-shaped and superimposed on top of the vehicle in the captured image. In some embodiments, the graphic indicator represents one or more semantics associated with the vehicle. For example, the graphic indicator can represent the light indicators of a vehicle identified in the captured image, such as whether the light indicators of the vehicle are identified (by the machine learning model 124). In another example, the graphic indicator can represent the type of light indicator identified by the machine learning model 124. The graphic indicator can also be used to annotate one or more semantics associated with the vehicle.

[0075] For example, an analyst may utilize the analyzed light indicator information of the vehicle to annotate a graphical indicator associated with the vehicle. The annotation may be implemented by user input to an interactive user interface that presents an image or video clip acquired from the vehicle. In some embodiments, the graphical indicator overlay component 338 overlays a specific graphical representation on the graphical indicator associated with the vehicle in which the inconsistency exists. For example, if there is an inconsistency in a vehicle in a captured image such that the machine learning model identifies that the vehicle's brake lights are flashing, while the actual brake lights are off, the graphical indicator overlay component 338 may overlay the graphical indicator on the vehicle with a specific graphical representation. The specific graphical representation may be any representation, such as, based on the color or shape of the graphical indicator, adding any annotations on or near the graphical indicator, and the like.

[0076] In some embodiments, the graphical representation of the indicator is two-dimensional. In these embodiments, the graphic indicator superposition component 338 can identify image pixels associated with the vehicle. Then, a two-dimensional frame can be generated and superimposed on the image of the vehicle. In some embodiments, the graphic indicator superposition component 338 can perform image segmentation on the captured image to identify the vehicle and any lights. For example, the graphic indicator superposition component 338 can segment the captured image into areas (e.g., pixel groups) associated with the vehicle and those areas that do not correspond to the vehicle. In some embodiments, the graphic indicator superposition component 338 generates a two-dimensional frame on the area associated with the vehicle. In some embodiments, after determining the area associated with the vehicle, the graphic indicator superposition component 338 can generate a three-dimensional volume and superimpose it on the area associated with the vehicle.

[0077] Figure 3B The light indicator display component 340 in can be configured to display the vehicle light indicators identified in the captured image. The analyst can execute instructions to the light indicator display component 340 to obtain a set of images (or video clips) captured in sequence, each of which contains a graphical indicator. In some embodiments, the light indicator display component 340 can be configured to request the machine learning model to identify or predict the light indicators of the vehicles contained in the captured image. After the machine learning model identifies the light indicators associated with the vehicles in the captured image, the light indicator display component 340 is configured to obtain the identified light indicators and display them on the graphical indicator associated with each vehicle in the captured image.

[0078] Figure 3B The machine learning model validation component 342 in can be configured to determine one or more captured images that have at least one inconsistency between the vehicle light indicators identified in the image and the actual light indicators. Using the machine learning model validation component 342, the analyst can detect the inconsistency by comparing the identified light indicators with the actual light indicators and the vehicle travel path in the image. Some examples of potential inconsistencies are shown in Table I.

[0079]

[0080] Table I

[0081] The memory 330 also includes a machine learning model training component 344, which is configured to train the machine learning model. The training can be based on inconsistency correction. In some embodiments, one or more vehicles associated with the detected inconsistency in the captured image are annotated with correct light indicators. The correct light indicator can be determined by an analyst. The analyst can annotate the correct light indicator information on the graphic indicator of the vehicle associated with the inconsistency. After receiving the annotation (correct light indicator data), the machine learning model training component 348 can correct the captured image with one or more inconsistencies and store them in the network server 128. In some embodiments, the processing unit 322 commands the machine learning model 124 to execute instructions to update, modify or add one or more attributes related to the light indicator identification based on the corrected image.

[0082] Figure 4 An example of a vehicle with a camera for capturing images of other vehicles is shown below. For illustration purposes, Figure 4 You can refer to Figure 1 , Figure 2 , Figure 3A and Figure 3B For illustration purposes, Figure 4 A top view including the vehicle 110 and surrounding vehicles captured by cameras 220, 230, and 240 is shown. The vehicle 110 may include a front camera 220, a pillar camera 230, and a relay camera 240. The front camera 220 is configured to capture a front image located in front of the vehicle 110, so that the vehicle 410 can be captured in the image. The pillar camera 230 is configured to capture a side image located on both sides of the vehicle 110, so that the vehicle 412 can be captured in the image. The relay camera 240 is configured to capture a rear image located behind the vehicle 110, so that the vehicle 414 can be captured in the image.

[0083] Figure 5An example of a captured image is illustrated, including an identified light indicator associated with each vehicle in the image. The illustrated example may be included in an interactive user interface used, for example, by a user associated with annotated training data. The user may access an image or video clip acquired from a fleet of vehicles executing the above-described machine learning model. Thus, the image forms part of a video clip acquired by a vehicle in the fleet of vehicles. Advantageously, in order to reduce the burden associated with annotation, the image may include a bounding box indicating an object (e.g., a vehicle) as well as a light indicator (e.g., light indicator 506). Light indicator 506 may be assigned by the machine learning model, and the user may provide user input (e.g., touch-based input, mouse / keyboard, voice input) to change light indicator 506 (e.g., so that the indicator reacts to a left turn, brake lights, hazard warning lights, etc.).

[0084] For ease of explanation, Figure 5 You can refer to Figure 1 , Figure 2 , Figure 3A and Figure 3B The light indicators can be identified by the machine learning model 124. Figure 5 As shown, image 500 (including vehicle 502) is captured by a front camera mounted on vehicle 110. The front camera of vehicle 110 can capture a front view image of the vehicle, and vehicle 110 is configured to transmit the captured image to network server 128. In some embodiments, a user (e.g., an analyst) using verification computing device 130 can view or overlay a graphical indicator on each vehicle included in the captured image.

[0085] For example, Figure 5 As shown, the graphic indicator 504 has a box shape and is superimposed on each vehicle 502. In one embodiment, the analyst uses the verification computing device 130 to superimpose the graphic indicator 504 on selected vehicles (such as several vehicles near the vehicle 110). The number of selected vehicles can be determined based on a specific application. In some embodiments, the graphic indicator 504 can include one or more semantics related to the vehicle. In this example, Figure 5 As shown, each graphical indicator 504 includes an identified light indicator 506 associated with the vehicle. In these embodiments, the light indicator of each vehicle can be identified by the machine learning model 124. The machine learning model 124 identifies the light indicator of each vehicle based on at least its learning parameters, algorithms, or attributes related to identifying the light indicators. For example, the light indicators can include turn signals, brake lights, emergency lights, etc.

[0086] FIG. 6A to FIG. 6BAn example of validation of a machine learning model in identifying light indicators of one or more surrounding vehicles is illustrated. Validation can be based on a comparison of vehicle light indicators identified in a captured image using the machine learning model with the actual light indicators and travel path of the vehicle. In some embodiments, one or more vehicles in a captured image are validated as having inconsistencies between the identified light indicators and the actual light indicators and travel path of the vehicle. In these embodiments, the inconsistency can be corrected by receiving one or more inputs from an analyst authorized to validate the machine learning model. For ease of illustration, FIG. 6A to FIG. 6B You can refer to Figure 1 , Figure 2 , Figure 3A and Figure 3B Let's discuss some of the components in it.

[0087] Fig. 6A An example of verification of the identified light indicators of a vehicle in captured image 600 is illustrated. In some embodiments, the light indicators identified based on the machine learning model can be verified by analyzing a set of sequentially captured images. For example, to verify the identified light indicators of a vehicle in an image, 300 images captured immediately after the image (played in a video clip for 10 seconds at 30fps) can be analyzed to determine the actual light indicators and travel path of the vehicle. The number of images (or video clip playback time) can be determined based on the specific application.

[0088] In addition, Fig. 6A , for illustrative purposes, two captured images are superimposed and used to determine the actual light indicators and travel paths of the vehicles contained in the images. For example, a first captured image includes a first group of vehicles 604, 606, 608, and a second captured image includes a second group of vehicles 614, 616, 620. The first captured image and the second captured image are captured sequentially, and the second captured image is captured immediately after the first captured image. In some embodiments, the identified light indicators 610, 620, 630 of the vehicles 604, 606, 608 are superimposed on top of the graphical indicator 602 associated with each vehicle. In these embodiments, the light indicators of the vehicles 604, 606, 608 are identified using the machine learning model 124. The identified light indicators of the vehicles 604, 606, 608 can be compared to the captured images including the actual light indicators.

[0089] In some embodiments, the actual light indicators may be determined by analyzing the travel paths of the vehicles 604, 606, 608. For example, the vehicle 614 included in the second captured image shows that the vehicle 604, which is identified as "brake lights on", is traveling in the forward direction without reducing its speed. In this example, the vehicle 604 may be verified as having an inconsistency between the identified signal indicator 610 and the actual signal indicator and travel path 614 of the vehicle. In another example, the vehicle 616 included in the second captured image may verify that the vehicle 606, which is identified as "light indicators off", is turning into the right lane 616 with a "flashing right turn signal". In this example, the vehicle 606 is determined to have an inconsistency between the identified signal indicator 620 and the actual signal indicator and travel path 616 of the vehicle. Finally, in another example, the vehicle 608, which is identified as "flashing left turn signal", is traveling in the forward direction 618 without a flashing left turn signal. In this example, vehicle 608 is determined to have an inconsistency between identified signal indicator 630 and the vehicle's actual signal indicator and travel path 618. Various types of inconsistencies may be detected in the system, various examples of which are described above in Table I.

[0090] Figure 6B An example of an interactive graphical user interface 650 is illustrated to annotate a graphical indicator of a vehicle in a captured image. For example, an analyst using the verification computing device 130 can access the interface 640 to correct vehicles 604, 606, 608 having light indicators 610, 620, 630 incorrectly identified by the machine learning model. The interactive graphical user interface can be provided as an application stored and executed by a processor of the verification computing device. In another example, the interactive graphical user interface can be provided by one or more processors implemented in the network server 128 so that a user uses network server resources to correct the light indicator detection of the vehicle. In some embodiments, the interface 604 displays whether each vehicle light is activated or inactivated. In some embodiments, a user who is authorized to annotate or modify a label associated with a vehicle light can modify the activation or inactivation state using a user input interface, such as a mouse, keyboard, etc.

[0091] Figure 7 An example of labeling the correct light indicator of the vehicle in the captured image 700 to the graphical indicator is shown. Figure 7 As shown, vehicles 604, 606, 608 (such as Fig. 6A ) are respectively modified by light indicators 710, 720, 730. For example, by superimposing light indicators 710, 720, 730 to modify (as shown in FIG. Fig. 6AIn some embodiments, the corrected image with the superimposed light indicator annotations is fed into a machine learning model to train the model. In some embodiments, the trained machine learning model can be fed into the autonomous driving system 210 (e.g., Figure 2 shown).

[0092] Figure 8 is a flow chart illustrating one embodiment of an example process that a verification computing device may perform to detect one or more vehicles in a captured image that have inconsistencies between identified light indicators and actual light indicators of the vehicles. According to this embodiment, Figure 8 The process shown in may include fewer or more blocks and / or the blocks may be performed in a different order than shown. For ease of explanation, Figure 8 The process can be referred to Figure 1 , Figure 2 , Figure 3B , Figure 4 and Figure 5 Let's discuss some of the components in it.

[0093] Starting from block 810, the verification computing device can obtain a captured image of the vehicle's surrounding perspective. The verification computing device can obtain the captured image by accessing a network server. In some embodiments, the verification computing device downloads the captured image. In other embodiments, the verification computing device can access the captured image using a virtual computing service resource provided by a machine learning training system. In some embodiments, the vehicle can use a front camera mounted on the front side of the vehicle, such as a windshield, to capture a front view image of the vehicle. The captured image can be formed into multiple video clips by merging a group of sequentially captured images. For example, every 300 sequentially captured images are merged into a video file that can be played at 30fps for about 10 seconds. The number of captured images and the specifications of the video clips (including frame rate and resolution) can be determined based on the specific application. In some embodiments, the vehicle is configured to be wirelessly connected to the network and the captured image is transmitted to the network server via the network. In some embodiments, the wireless standard for connecting the network can be based on high-speed 4G LTE or other wireless communication technologies, such as 5G communication. Therefore, in some embodiments, the network may include one or more wireless networks, such as a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a long-term evolution (LTE) network, or any other type of wireless network. The network may use protocols and components to communicate over the Internet or any network of the above types. For example, the protocols used by the network may include Hypertext Transfer Protocol (HTTP), HTTP Security (HTTPS), Message Queuing Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), etc. The protocols and components for communicating over the Internet or the above types of communication networks are known to those skilled in the art and are therefore not described in detail herein.

[0094] Proceeding to block 820, the verification computing device can generate a graphical indicator for each vehicle and overlay the graphical indicator on each associated vehicle. The graphical indicator can be box-shaped and overlaid on top of the vehicle in the captured image. In one embodiment, the verification computing device overlays the graphical indicator on selected vehicles in the captured image, such as certain vehicles that are closer to the vehicle that captured the image. The number of selected vehicles can be determined based on the specific application. In some embodiments, the graphical indicator represents one or more semantics associated with the vehicle.

[0095] In some embodiments, the graphical representation of the indicator is two-dimensional. In these embodiments, the verification computing device can identify image pixels associated with the vehicle. Then, a two-dimensional box (e.g., a bounding box) can be generated and superimposed on the image of the vehicle. In some embodiments, the verification computing device can perform image segmentation on the captured image to identify the vehicle and any lights. For example, the verification computing device can segment the captured image into areas (e.g., groups of pixels) associated with the vehicle and those areas that do not correspond to the vehicle. In some embodiments, the verification computing device generates a two-dimensional box (e.g., a bounding box) on the area associated with the vehicle. In some embodiments, after determining the area associated with the vehicle, the graphic indicator overlay component 338 can generate a three-dimensional volume and overlay it on the area associated with the vehicle.

[0096] Proceeding to block 830, in some embodiments, the machine learning model identifies the light indicators of the vehicle in the captured image. The identified light indicators can be fed into a verification computing device.

[0097] Proceeding to block 840, the verification computing device may display the identified light indicators associated with each vehicle on the graphical identifier of the vehicle in the captured image. For example, the graphical indicator may indicate whether the machine learning model identified the light indicator of the vehicle. In some embodiments, the graphical indicator may indicate the type of light indicator identified by the machine learning model.

[0098] Proceeding to block 850, the verification computing device can detect an inconsistency between the vehicle's identified light indicators and the vehicle's actual light indicators. The inconsistency can be detected by analyzing a set of images captured in sequence. For example, to verify the vehicle's identified light indicators in an image, 300 images captured immediately after the image (played in a video clip for 10 seconds at 30fps) can be analyzed to determine the vehicle's actual light indicators and travel path. The number of images (or video clip playback time) can be determined based on the specific application. In some embodiments, the inconsistency type is a false positive or a false negative. However, various types of inconsistencies can be detected in the system, and various examples of inconsistencies are described in Table I above.

[0099] Proceeding to block 860, when it is determined at block 850 that one or more vehicles in the captured image are detected as having an inconsistency between the identified light indicators and the actual light indicators, the verification computing device can mark the image and store it in the network server. The verification computing device can also store the marked image to its internal or external storage medium. In some embodiments, the stored marked image including the inconsistency is used to train the machine learning model.

[0100] Upon determining at block 850 that no inconsistencies exist, the verification computing device may end the process of detecting inconsistencies.

[0101] Fig. 9 is a flow chart illustrating one embodiment of an example process that a verification computing device may perform to train a machine learning model. According to this embodiment, Fig. 9 The process shown in the may include fewer or more blocks and / or the blocks may be performed in a different order than shown. For ease of explanation, Fig. 9 The process can be referred to Figure 1 , Figure 2 , Figure 3A , Figure 3B and Fig. 6A Let's discuss some of the components in it.

[0102] Starting from block 910, the verification computing device can obtain the tagged image by accessing the network server. Each of the tagged images or the video clips including the tagged images can contain one or more vehicles for which there is an inconsistency between the identified light indicator and the actual light indicator. In some embodiments, the inconsistency type is a false positive or a false negative.

[0103] Moving to block 920, the verification computing device may request an analyst to correct the light indicators of the vehicles with inconsistencies. The analyst may be an authorized user who has the authority to verify the machine learning model, including a manager, developer, supervisor, administrator, etc.

[0104] Moving to block 930, upon receiving the request, the analyst may correct the light indicators of the vehicle with the inconsistency. Moving to block 940, in some embodiments, the analyst may annotate the graphical indicator of the vehicle with the inconsistency. The annotation may include the correct light indicators of the vehicle. In some embodiments, the annotated image with the correct light indicators is superimposed on the original image with the vehicle with the inconsistency. The annotated image may be stored in a network server.

[0105] Moving to block 950, the verification computing device can transmit the annotated image containing the correct light indicators of the vehicle to the machine learning model. In these embodiments, the machine learning model receives the annotated image with the correct light indicators and trains the machine learning module. For example, the parameters of the machine learning model can be updated (e.g., by gradient descent).

[0106] Moving to block 960, in some embodiments, the trained machine learning model can be fed into the vehicle's autonomous driving system. The vehicle can access and download the trained machine learning model via a network.

[0107] FIG. 10A to FIG. 10DExamples of light indicator detection in different environments are illustrated. This example user interface is for illustrative purposes only to illustrate various functions of the system. As previously mentioned, the surrounding image of the vehicle is captured using a camera mounted on the vehicle. The autonomous driving system can determine the light indicators by utilizing a machine learning model and display them on the detected vehicle contained in the captured image.

[0108] Fig. 10A is an example of light indicator detection on a high-density road. In a high-density road environment, the vehicle can detect most nearby vehicles and determine the light indicators of the detected vehicles that are closer to the vehicle. For example, vehicle 110 can determine the light indicators of the approaching vehicle 1002.

[0109] Fig. 10B 1014; oncoming vehicles 1016; and parked vehicles 1018. The priorities discussed herein are examples only of examples, and the priorities are not limited thereto.

[0110] Fig. 10C is an example of light indicator detection in a parking lot. As shown in the example, the vehicle can capture an image of a parking lot and determine the light indicators of the parking lot vehicle 1020.

[0111] Fig. 10D is an example of light indicator detection for various types of moving objects. Fig. 10D As shown, analysts can determine light indicators of various types of moving objects, including but not limited to motorcycles 1032, buses 1034, and any type of vehicle 1036. In some embodiments, analysts can determine light indicators by detecting lights on the edge of vehicle 1038.

[0112] Fig.11 An example interactive user interface 1100 is illustrated that a user (e.g., an analyst) may use. The interactive user interface 1100 is presenting images from image sensors or cameras disposed around the vehicle. As described herein, the vehicle may provide images or video clips to the system to update the machine learning model. In the example shown, the images thus reflect images from these cameras or image sensors. Although images are illustrated, it is understood that the images may form a video clip, and the user may select to play the video clip or a selected portion thereof.

[0113] The user interface 1100 includes a first image 1102 having a bounding box 1104 about an object (e.g., a truck). As shown, the object is contained in multiple images from different cameras. Located near the bounding box 1104 is a light indicator 1106 (e.g., a graphical indicator of the light indicator), which in this example is a graphical icon (e.g., a hand pointing to the left, indicating a left turn signal). There can be multiple graphical icons, providing a convenient way for the user to understand whether the light indicator 1106 is a left turn signal, a right turn signal, a hazard warning light, a brake light, etc.

[0114] As described herein, the light indicator 1106 may be determined by a machine learning model executed on the vehicle. Figures 1 to 10D , annotation information can be provided with the image or video clip (at least indicating annotations associated with lights on the vehicle). As another example, a system (e.g., presenting a user interface or analyzing an image or video clip and performing training) can execute a machine learning model to determine the annotations.

[0115] Light indicator 1106 may be presented during presentation of the video clip near bounding box 1104. For example, light indicator 1106 may be presented at a similar offset from bounding box 1104 so that it fits with bounding box 1104. Similarly, if the object has a trailer attached, light indicator 1106 may be presented at a similar offset from the trailer.

[0116] A user of user interface 1102 may provide user input to update light indicator 1106. For example, the user may select indicator 1106 and a drop-down menu or other user interface (e.g., Figure 6B 1106. In this way, the user can generate a ground truth (e.g., updated indicator 1106) for training a machine learning model.

[0117] The user interface 1100 also includes a progress bar 1108 that enables selection of different portions of the video clip. For example, the progress bar 1108 may extend from a first timestamp to a final timestamp. In some embodiments, a portion of the progress bar 1108 may be a first color (e.g., green) indicating that there are no errors or problems associated with the light indicator. The progress bar 1108 may be a second color (e.g., red) indicating that the user has updated the light indicator.

[0118] Various embodiments of the present disclosure may be systems, methods and / or computer program products at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present disclosure.

[0119] For example, the functions described herein may be performed as software instructions executed by one or more hardware processors and / or any other suitable computing devices and / or in response to software instructions being executed by one or more hardware processors and / or any other suitable computing devices. The software instructions and / or other executable code may be read from a computer-readable storage medium (or multiple media).

[0120] A computer-readable storage medium may be a tangible device capable of retaining and storing data and / or instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device (including any volatile and / or non-volatile electronic storage device), a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer floppy disk, a hard disk, a solid-state drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device having instructions recorded thereon, a protruding structure in a punch card or a groove, and any suitable combination of the above. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0121] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0122] Computer-readable program instructions (also referred to herein as, for example, "code," "instructions," "modules," "applications," "software applications," etc.) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, C++, etc.) and procedural programming languages ​​(e.g., "C" programming language or similar programming languages). Computer-readable program instructions may be called from other instructions or themselves, and / or may be called in response to detected events or interrupts. Computer-readable program instructions configured to be executed on a computing device may be provided on a computer-readable storage medium, and / or provided as a digital download (and may be initially stored in a compressed or installable format, requiring installation, decompression, or decryption prior to execution), which may then be stored on a computer-readable storage medium. Such computer-readable program instructions may be stored in part or in whole on a storage device (e.g., a computer-readable storage medium) executing a computing device for execution by the computing device. The computer-readable program instructions may be executed entirely on a user's computer (e.g., an execution computing device), partially on a user's computer, as a standalone software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuit devices, including, for example, programmable logic circuit devices, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may be personalized electronic circuit devices by utilizing state information of computer-readable program instructions, thereby executing computer-readable program instructions to perform aspects of the present disclosure.

[0123] Aspects of the present disclosure are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each block in the flow charts and / or block diagrams and the combination of blocks in the flow charts and / or block diagrams can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a means for implementing the functions / actions specified in the flowchart and / or block diagram(s). These computer-readable program instructions may also be stored in a computer-readable storage medium, which may instruct a computer, a programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing aspects of the functions / actions specified in the flowchart(s) and / or block diagram(s).

[0125] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device, so that a series of operating steps are performed on the computer, other programmable device, or other device, thereby generating a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other device implement the functions / actions specified in the flowchart and / or block diagram (multiple) blocks. For example, the instructions can initially be carried on a disk or solid-state drive of a remote computer. The remote computer can load instructions and / or modules into its dynamic memory and send instructions via a telephone, cable, or optical cable using a modem. A modem local to the server computing system can receive data on the telephone / cable / optical cable and place the data on the bus using a converter device including an appropriate circuit device. The bus can transfer data to a memory, from which the processor can retrieve and execute instructions. The instructions received by the memory can optionally be stored on a storage device (e.g., a solid-state drive) before or after execution by the computer processor.

[0126] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flow chart or block diagram can represent the part of a module, segment or instruction, which includes one or more executable instructions for realizing (multiple) specified logical functions. In some alternative implementations, the function pointed out in the block may not occur in the order pointed out in the figure. For example, two blocks shown in succession can actually be performed substantially at the same time, or blocks can sometimes be performed in reverse order, depending on the functions involved. In addition, some blocks can be omitted in some implementations. The method and process described herein are also not limited to any particular order, and the blocks or states related thereto can be performed in appropriate other orders.

[0127] It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, can be implemented by a dedicated hardware system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions. For example, any process, method, algorithm, element, block, application, or other function (or portion of a function) described in the preceding section can be embodied in electronic hardware and / or fully or partially automated by electronic hardware, such as a dedicated processor (e.g., an application specific integrated circuit (ASIC)), a programmable processor (e.g., a field programmable gate array (FPGA)), a dedicated circuit device, and / or the like (any of which may also combine custom hardwired logic, logic circuits, ASICs, FPGAs, etc. with custom programming / execution of software instructions to implement these techniques).

[0128] Any of the above processors and / or devices including any of the above processors may be referred to herein as, for example, a "computer", "computer device", "computing device", "hardware computing device", "hardware processor", "processing unit", etc. The computing devices of the above embodiments may typically (but not necessarily) be controlled and / or coordinated by operating system software, such as Mac OS, iOS, Android, Chrome OS, Windows OS (e.g., Windows XP, Windows Vista, Windows 7, Windows 8, Windows 10, Windows Server, etc.), Windows CE, Unix, Linux, SunOS, Solaris, Blackberry OS, VxWorks, or other suitable operating systems. In other embodiments, the computing device may be controlled by a proprietary operating system. Conventional operating systems for execution control and schedule computer processes, perform memory management, provide file system, network, I / O services, and provide user interface functions, such as a graphical user interface ("GUI"), etc.

[0129] As described above, in various embodiments, users can access certain functions through a web-based viewer (e.g., a web browser) or other suitable software programs. In such implementations, the user interface can be generated by a server computing system and transmitted to the user's web browser (e.g., running on the user's computing system). Alternatively, the server computing system can provide the data required to generate the user interface (e.g., user interface data) to the browser, and the user interface can be generated in the browser (e.g., the user interface data can be executed by a browser accessing a web service, and can be configured to present the user interface based on the user interface data). Then, the user can interact with the user interface through a web browser. The user interface of certain implementations can be accessed through one or more dedicated software applications. In certain embodiments, one or more of the computing devices and / or systems of the present disclosure may include a mobile computing device, and the user interface can be accessed through such mobile computing devices (e.g., smart phones and / or tablet computers).

[0130] Many changes and modifications may be made to the above-described embodiments, and the elements of these changes and modifications should be understood as other acceptable examples. All of these modifications and variations are intended to be included within the scope of the present disclosure. The foregoing description details certain embodiments. However, it should be understood that no matter how detailed the foregoing text appears, the systems and methods can be practiced in a variety of ways. As described above, it should be noted that the use of specific terms in describing certain features or aspects of the systems and methods should not be understood as implying that the terms are redefined herein to include any specific characteristics of the features or aspects of the systems and methods associated with the terms.

[0131] Conditional language, such as "can," "may," "might," or "might," etc., unless expressly stated otherwise or otherwise understood in the context of use, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments may not include these features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic (with or without user input or prompting) for determining whether these features, elements, and / or steps are included or performed in any particular embodiment.

[0132] Unless expressly stated otherwise, linking language such as the phrase "at least one of X, Y, and Z" or "at least one of X, Y, or Z" should be understood in context and is generally used to express that an item, term, etc. can be X, Y, or Z, or a combination thereof. For example, the term "or" is used in an inclusive sense (rather than an exclusive sense), and thus, when used to link a list of elements, the term "or" means one, some, or all of the elements in the list. Therefore, such linking language is generally not intended to imply that certain embodiments require that at least one of X, at least one of Y, and at least one of Z each be present.

[0133] The terms "a" and "an" as used herein should be given an inclusive rather than exclusive interpretation. For example, unless otherwise specified, the terms "a" and "an" should not be understood as "exactly one" or "one and only one"; instead, the terms "a" and "an" mean "one or more" or "at least one", whether used in the claims or elsewhere in the specification, and regardless of the use of quantifiers such as "at least one", "one or more" or "a plurality" in the claims or elsewhere in the specification.

[0134] The term "comprising" as used herein should be given an inclusive rather than exclusive interpretation. For example, a general purpose computer including one or more processors should not be interpreted to exclude other computer components and may include components such as memory, input / output devices and / or network interfaces.

[0135] Although the foregoing detailed description has shown, described and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions and changes in the form and details of the devices or processes shown may be made without departing from the spirit of the present disclosure. It will be recognized that certain embodiments of the inventions described herein may be embodied in a form that does not provide all of the features and advantages described herein, as certain features may be used or practiced separately from other features. The scope of certain inventions disclosed herein is indicated by the appended claims rather than the foregoing description. All changes that fall within the meaning and scope of equivalents of the claims are intended to be included within their scope.

Claims

1. A system for annotating images to train a machine learning model to detect light indicators on a vehicle, the system comprising one or more processors and a non-transitory computer storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: acquiring an image of one or more vehicles on a road; identifying a location of each of the one or more vehicles; displaying, via a user interface, a graphical indicator on each of the one or more vehicles to indicate that the vehicle is detected by the system; as well as Receiving, via the user interface, an indication of whether a light indicator is activated or inactivated on each of the one or more vehicles to annotate the image for the machine learning model.

2. The system of claim 1 , wherein acquiring an image comprises: Acquire images from multiple vehicles with autonomous driving systems.

3. The system of claim 2, wherein acquiring an image comprises: When the automated driving system determines that the light indicator detection is incorrectly determined by the automated driving system, images of the plurality of vehicles are acquired.

4. The system of claim 1 , wherein identifying the location of each of the one or more vehicles comprises: A vehicle in the image is identified, and graphical coordinates of the vehicle in the image are determined.

5. The system of claim 1 , wherein displaying the graphical indicator on each of the one or more vehicles comprises: A bounding box is displayed around each of the one or more vehicles in the acquired image.

6. The system of claim 1 , wherein identifying the location of each of the one or more vehicles comprises: Image segmentation is performed on the acquired images, and wherein the image segmentation generates a region of each acquired image corresponding to the vehicle.

7. The system of claim 1 , wherein receiving an indication of whether the light indicator is activated or inactivated comprises: A mouse selection is received from a user, the mouse selection labeling the vehicle as having activated light indicators or inactivated light indicators.

8. The system of claim 1 , wherein receiving the indication as to whether a light indicator is activated or inactivated comprises: Receive an indication as to whether the brake lights are activated or inactivated.

9. The system of claim 1 , wherein receiving the indication as to whether a light indicator is activated or inactivated comprises: An indication is received as to whether the turn signal is activated or inactivated.

10. A system for annotating images to train a machine learning model to detect light indicators on a vehicle, the system comprising one or more processors and a non-transitory computer storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: acquiring an image of one or more vehicles on a road; identifying a location of each of the one or more vehicles in the acquired image; determining whether a light indicator is indicated as active or inactive by an automated driving system in each of the one or more vehicles; determining from the images of the one or more vehicles that one or more vehicles have a false prediction of whether the light indicator is activated or inactivated; as well as The images having false predictions are annotated through a user interface with a correct indication as to whether the light indicator is activated or inactivated.

11. The system of claim 10, wherein identifying the location of each of the one or more vehicles comprises: A vehicle in the image is identified, and graphical coordinates of the vehicle in the image are determined.

12. The system of claim 10, wherein acquiring an image comprises: Acquire images from multiple vehicles with autonomous driving systems.

13. The system of claim 12, wherein acquiring an image comprises: When the automated driving system determines that the light indicator detection is incorrectly determined by the automated driving system, images from the plurality of vehicles are acquired.

14. The system of claim 10, further comprising: displaying, via the user interface, a graphical indicator on each of the one or more vehicles to indicate that the vehicle is detected by the system; 15. The system of claim 14, wherein displaying the graphical indicator on each of the one or more vehicles comprises: A bounding box is displayed in the acquired image around each of the one or more vehicles.

16. The system of claim 10, wherein the indication of whether the light indicator of each of the one or more vehicles is activated or inactivated is predicted by an autonomous driving system of each of the vehicles.

17. The system of claim 10, wherein the false prediction is a discrepancy between the light indicator and the position of the vehicle.

18. The system of claim 10, further comprising: An updated light indicator is received, and a mouse selection is received from a user, the mouse selection annotating the vehicle with the light indicator based on the location of the vehicle.

19. The system of claim 10, wherein the indication as to whether a light indicator is activated or inactivated is an indication as to whether a brake light is activated or inactivated.

20. The system of claim 10, wherein the indication as to whether a light indicator is active or inactive is an indication as to whether a turn signal is active or inactive.

21. A method for annotating images to train a machine learning model to detect light indicators on a vehicle, the method comprising: acquiring an image of one or more vehicles on a road; identifying a location of each of the one or more vehicles; annotating, via the user interface, an indication as to whether a light indicator is activated or inactivated on each of the one or more vehicles; determining one or more vehicles having false predictions from the images of the one or more vehicles; An updated indication is received regarding whether the light indicator is active or inactive on the vehicle having the false prediction.

22. The method of claim 21 , wherein identifying the location of each of the one or more vehicles comprises: A vehicle in the image is identified, and graphical coordinates of the vehicle in the image are determined.

23. The method of claim 21, wherein acquiring an image comprises: Acquire images from multiple vehicles with autonomous driving systems.

24. The method of claim 21, wherein acquiring an image comprises: When the light indicator detection is incorrectly determined by the autonomous driving system of each vehicle, images from the one or more vehicles are acquired.

25. The method of claim 21, further comprising: Displaying a graphical indicator on each of the one or more vehicles to indicate that the vehicle was detected by the machine learning model.

26. The method of claim 25, wherein displaying the graphical indicator on each of the one or more vehicles comprises: A bounding box is displayed around each of the one or more vehicles in the acquired image.

27. The method of claim 21, wherein the indication of whether the light indicator of each of the one or more vehicles is activated or inactivated is predicted by an autonomous driving system of each of the vehicles.

28. The method of claim 21, wherein the false prediction is a discrepancy between the light indicator and the position of the vehicle.

29. The method of claim 21, wherein receiving an updated light indicator comprises: A mouse selection is received from a user, the mouse selection labeling the vehicle with the light indicator based on the position of the vehicle.

Images