Determine whether image data is within a predetermined range configured for image analysis software to analyze

Abstract

A system for determining whether image data falls within a predetermined range configured for analysis by image analysis software. The system includes a camera and an electronic processor. The electronic processor is configured to receive image data from the camera and generate a prediction regarding the image data and a confidence value associated with the prediction. The electronic processor is also configured to perturb the image data using perturbation values ​​and generate a prediction regarding the perturbed image data and a confidence value associated with the prediction. The electronic processor is further configured to compare the confidence values ​​and disable the autonomous functions of the vehicle when the difference between the confidence value associated with the prediction regarding the image data and the confidence value associated with the prediction regarding the image data is less than a predetermined threshold value.

Description

Technology Field

[0001] This application claims priority to U.S. provisional patent application No. 62 / 786,590 filed on December 31, 2018, the entire contents of which are incorporated by reference.

[0002] Image sensors are important components in autonomous and partially autonomous driving systems. However, in some situations, image data from image sensors may be analyzed inaccurately. For example, in situations such as adverse weather conditions (e.g., rain), low light, and direct sunlight, image analysis software may generate inaccurate predictions regarding the presence of objects in the image, the location of objects in the image, the presence of road markings in the image, the location of road markings in the image, or a combination thereof. The reason the image analysis software makes inaccurate decisions in some situations is that the image analysis software is not configured to analyze image data acquired in these situations. Therefore, in some embodiments, it is desirable that image data is ignored when image data is acquired from a camera in a situation where the image analysis software is not configured to analyze it. In some embodiments, it is desirable that at least some autonomous functions of the vehicle are disabled when image data is acquired from a camera in a situation where the image analysis software is not configured to analyze it. The embodiments of the present invention describe a method and a system for detecting whether image data has been acquired in a situation where the image analysis software is not configured to analyze it.

[0003] One embodiment provides a system for determining whether image data is within a predetermined range configured for image analysis software to analyze. The system includes a camera and an electronic processor. The electronic processor is configured to receive image data from the camera and, using image analysis software, generate a prediction regarding the image data and a confidence value associated with the prediction regarding the image data. The electronic processor is also configured to perturb the image data using a perturbation value and, using image analysis software, generate a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data. The electronic processor is further configured to compare the confidence values ​​associated with the prediction regarding the perturbed image data with the confidence values ​​associated with the prediction regarding the image data and to disable the autonomous function of the vehicle when the difference between the confidence values ​​associated with the prediction regarding the image data and the confidence values ​​associated with the prediction regarding the image data is less than a predetermined threshold value.

[0004] Another embodiment provides a method for determining whether image data is within a predetermined range configured for image analysis software to analyze. The method includes the steps of receiving image data from a camera using an electronic processor, and using image analysis software to generate a prediction regarding the image data and a confidence value associated with the prediction regarding the image data. The method also includes the steps of perturbing the image data using a perturbation value, and using image analysis software to generate a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data. The method further includes the step of using an electronic processor to compare the confidence value associated with the prediction regarding the perturbed image data with the confidence value associated with the prediction regarding the image data. The method also includes the step of using data from alternate sensors to generate predictions for controlling the autonomous functions of a vehicle when the difference between the confidence value associated with the prediction regarding the image data and the confidence value associated with the prediction regarding the image data is less than a predetermined threshold value.

[0005] Other aspects, features, and embodiments will become apparent from the detailed description and the accompanying drawings. Brief explanation of the drawing

[0006] FIG. 1 is a block diagram of a system for determining whether image data is within a predetermined range configured for image analysis software to analyze, according to some embodiments. FIG. 2 is a block diagram of an electronic controller of the system of FIG. 1 according to some embodiments. FIG. 3 is a flowchart of a method using the system of FIG. 1 to determine whether image data is within a predetermined range configured for image analysis software to analyze, according to some embodiments. Specific details for implementing the invention

[0007] Before any embodiments are described in detail, it should be understood that the present disclosure is not intended to be limited to the details of the configurations and arrangements of components described in the following description or illustrated in the following drawings in the application. Other configurations may be possible and the embodiments may be implemented or performed in various ways.

[0008] Multiple hardware and software-based devices and multiple different structural components may be used to implement various embodiments. Additionally, embodiments may include hardware, software, and electronic components or modules, which may be illustrated and described as if most components are implemented in hardware only for the purposes of discussion. However, a person skilled in the art will recognize, based on reading this detailed description, that in at least one embodiment, electronic-based aspects of the invention may be implemented in software executable by one or more processes (e.g., stored on a non-transient computer-readable medium). For example, the “control units” and “controllers” described herein may include one or more electronic processors, one or more memory modules including a non-transient computer-readable medium, one or more input / output interfaces, one or more application-specific integrated circuits (ASICs), and various connections connecting the various components (e.g., system buses).

[0009] FIG. 1 illustrates a system (100) for determining whether image data is within a predetermined range configured for image analysis software to analyze. The system (100) includes a vehicle (105). Although the vehicle (105) is illustrated as a four-wheeled vehicle, it may include vehicles of various types and designs. For example, the vehicle (105) may be an automobile, a motorcycle, a truck, a bus, a semi-tractor, etc. In the illustrated example, the vehicle (105) includes an electronic controller (110), a camera (115), a steering system (120), an accelerator (125), and brakes (130). The components of the vehicle (105) may have various configurations and may use various communication types and protocols.

[0010] The electronic controller (110) may be communicably connected to the camera (115), steering system (120), accelerator (125), and brakes (130) via various wired or wireless connections. For example, in some embodiments, the electronic controller (110) is directly coupled to each of the aforementioned components of the vehicle (105) via a dedicated wire. In other embodiments, the electronic controller (110) is communicably coupled to one or more of the components via a shared communication link, such as a vehicle communication bus (e.g., a CAN (Controller Area Network) bus) or a wireless connection.

[0011] Each component of the vehicle (105) can communicate with the electronic controller (110) using various communication protocols. The embodiment illustrated in FIG. 1 provides an example of the components and connections of the vehicle (105). However, these components and connections may be configured in ways different from those illustrated and described herein. For example, it should be understood that the electronic controller (110) may include more than one camera than the single camera (115) illustrated in FIG. 1, and that the cameras included in the vehicle (105) may be installed at various locations inside and outside the vehicle (105).

[0012] FIG. 2 is a block diagram of an electronic controller (110) of a vehicle (105). The electronic controller (110) includes a plurality of electrical and electronic components that provide power, operation control, and protection to components and modules within the electronic controller (110). The electronic controller (110) particularly includes an electronic processor (200) (e.g., a programmable electronic microprocessor, a microcontroller, or a similar device), a memory (205) (e.g., a non-transient, machine-readable memory), and an input / output interface (210). The electronic processor (200) is communically connected to the memory (205) and the input / output interface (210). In cooperation with the memory (205) and the input / output interface (210), the electronic processor (200) is configured to implement the methods particularly described herein.

[0013] As will be described in more detail below, memory (205) includes computer-executable instructions (or software) for determining whether image data, in particular, is within a predetermined range (acquired in the situation) to which image analysis software is trained to analyze. In the example illustrated in FIG. 2. Memory (205) includes image analysis software (215) comprising a convolutional neural network (CNN) (220) (or more broadly machine learning software) and autonomous function software (225). The CNN (220) is trained to perform predictions related to detecting and classifying objects, road markings, road signs, etc., in image data received from the camera (115). The autonomous function software (225) relies on predictions made by the image analysis software (215) (e.g., CNN (220)) using image data from the camera (115) to provide autonomous functions to the vehicle (105), particularly by controlling the steering system (120), accelerator (125), and brakes (130). The autonomous function software (225) may be, for example, an automatic cruise control (ACC), an automatic braking system, an automatic parking system, etc. It should be understood that the memory (205) may contain more, fewer, or different software components than illustrated in FIG. 2. Although described herein as including a CNN, it should be noted that the image analysis software (215) may include different types of machine learning software, for example, a decision tree or a Bayesian network. In some embodiments, the image analysis software (215) may not include machine learning software.

[0014] The electronic controller (110) may be implemented as several independent controllers (e.g., programmable electronic controllers) each configured to perform specific functions or sub-functions. Additionally, the electronic controller (110) may include sub-modules comprising additional electronic processors, memory, or application-specific integrated circuits (ASICs) for input / output functions, processing of signals, and application of the methods listed below. In other embodiments, the electronic controller (110) includes additional, fewer, or different components.

[0015] FIG. 3 illustrates an example of a method (300) for determining whether image data is within a predetermined range configured for image analysis software to analyze. The method (300) is performed by an electronic processor (200) running image analysis software (215). Although an exemplary method (300) is described below in relation to a CNN trained to analyze image data, it should be understood that the method (300) is more generally applicable to image analysis software configured to analyze image data. In step (305), the electronic processor (200) receives image data from a camera (115). In step (310), the electronic processor (200) calculates a perturbation value specific to the CNN (220) and the received image data. A perturbation value is a value that effectively generates noise in the image data by changing the image data when added to the image data. The electronic processor (200) calculates the perturbation value using the gradient of the cost function of the CNN (220). In some embodiments, the cost function is defined as L(θ, x, y), where x is the input image data, y is the possible classifications for the image data, and θ is the value of the weights included in the CNN (220). The perturbation value is the cost function (▽) multiplied by the weight (ε). xThe sign (positive or negative) of the slope of L(θ, x, y) is determined as a result. In step (315), the image data is perturbed using a perturbation value (the perturbation value is added to each pixel of the image data). In step (320), the electronic processor (200) runs a CNN (220) to generate a prediction for the perturbed image data (classifying the perturbed image data into one of a plurality of categories). For example, the CNN (220) can determine whether the lane marking on the right side of the vehicle (105) is a solid line or a dashed line. When the electronic processor (200) generates a prediction, the prediction is associated with a confidence value. The confidence value indicates the probability that the prediction is accurate.

[0016] In step (325), the electronic processor (200) also runs the CNN (220) to generate a prediction regarding the image data (classifying the image data into one of a plurality of categories) and determines a confidence value associated with the prediction regarding the image data. Although step (325) is illustrated in FIG. 3 as being performed in parallel with steps (310-320), it should be noted that in some embodiments, these steps may be performed sequentially.

[0017] In step (330), the electronic processor (200) compares the confidence value associated with the prediction made based on the perturbed image data with the confidence value associated with the prediction made based on the unperturbed image data. In step (335), when the difference between the confidence values ​​is less than a predetermined threshold value, the electronic processor (200) determines that the image data received from the camera (115) is outside the predetermined range to which the CNN (220) is trained to analyze. When the image data received from the camera (115) is outside the predetermined range to which the CNN (220) is trained to analyze, the electronic processor (200) determines that the prediction made by the CNN (220) is unreliable. In some embodiments, when image data received from the camera (115) is outside a predetermined range trained to be analyzed by the CNN (220) and the autonomous function software (225) relies on predictions generated by the CNN (220), the electronic processor (200) disables the autonomous function of the vehicle (105) controlled by the autonomous function software (225).

[0018] In other embodiments, instead of disabling the autonomous function software (225) when image data received from the camera (115) is outside a predetermined range trained for analysis by the CNN (220), the electronic processor (200) ignores (disregards) the image data from the camera (115). When the electronic processor (200) ignores the image data from the camera (115), the electronic processor (200) uses data from alternative sensors to generate predictions with the CNN (220) (or other type of software) and executes the autonomous function of the autonomous function software (225). Examples of data from alternative sensors include image data from other cameras or data from sensors such as radar sensors, LIDAR sensors, ultrasonic sensors, etc.

[0019] It should be understood that the image analysis software (215) may include more than one CNN and each CNN may be trained to detect different things in the received image data. For example, one CNN may be trained to detect road markings (e.g., lane markings) and another CNN may be trained to detect objects such as people, animals, and vehicles. In some embodiments, when it is determined that the image data is outside the predetermined range to which the CNN is trained to analyze, the autonomous function controlled by the autonomous function software (225) is disabled, regardless of whether the autonomous function software (225) relies on predictions that the CNN makes based on the image data.

[0020] In the foregoing specification, specific embodiments have been described. However, those skilled in the art recognize that various modifications and changes may be made without departing from the scope of the invention as set forth in the following claims. Accordingly, the specification and drawings should be understood as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the teachings herein.

[0021] In this document, relational terms such as first and second, top and bottom, etc., may be used solely to distinguish one entity or action from another entity or action and do not necessarily require or imply any actual relationship or order between these entities or actions. The terms “comprises,” “includes,” “contains,” “comprising,” “including,” “containing,” “has,” “having,” or any other variations thereof are intended to cover non-comprehensive inclusion, so that a process, method, article, or device comprising (having) a list of elements may not include only these elements but may include other elements not explicitly described or unique to such process, method, article, or device. Without further restriction, the elements proceeding by “comprising” or “having” do not exclude the presence of additional identical elements in the process, method, article, or device comprising (having). The singular is defined as one or more unless expressly otherwise stated herein. The terms “substantially,” “essentially,” “approximately,” “about,” or any other version thereof are defined as similar to how a person skilled in the art would understand them, and in one non-limiting embodiment, the term is defined as within 10%, in another embodiment within 5%, in another embodiment within 1%, in another embodiment It is defined as within 0.5%. As used herein, the term “combined” is defined as connected, though not necessarily directly and not necessarily mechanically. A device or structure “configured” in a given manner may also be configured in ways not described, at least in such a manner.

[0022] Various features, advantages, and embodiments are described in the following claims.

Claims

Claim 1 A system for determining whether image data is within a predetermined range configured for image analysis software to analyze, comprising: a camera; and an electronic processor, wherein the electronic processor: receives the image data from the camera; uses the image analysis software to generate a prediction regarding the image data and a confidence value associated with the prediction regarding the image data; and a cost function of the image analysis software used to generate the prediction regarding the image data. The system is configured to determine the sign of the gradient of and calculate perturbation values ​​by multiplying the determined sign by a weight for setting the magnitude of the perturbation values, wherein x is input image data, y is possible classifications for the image data, and θ is the value of the weights included in the image analysis software used to generate the prediction; perturb the image data using the calculated perturbation values; generate a prediction regarding the perturbated image data and a confidence value associated with the prediction regarding the perturbated image data using the image analysis software; compare the confidence value associated with the prediction regarding the perturbated image data with the confidence value associated with the prediction regarding the image data; and determine that the image data received from the camera is within the predetermined range when the difference between the confidence value associated with the prediction regarding the perturbated image data and the confidence value associated with the prediction regarding the image data is greater than or equal to a predetermined threshold value, wherein the image analysis software includes machine learning software trained to analyze the image data, and the machine learning software includes a convolutional neural network. A system for determining whether image data is within a predetermined range configured for image analysis software to analyze. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 In claim 1, the electronic processor is configured to disable the autonomous function when the image data is outside the predetermined range and the autonomous function of the vehicle relies on the prediction regarding the image data, and the system determines whether the image data is within a predetermined range configured to be analyzed by image analysis software. Claim 6 delete Claim 7 A system for determining whether image data is within a predetermined range configured for analysis by the image analysis software, wherein the predictions generated by the image analysis software are related to detecting and classifying objects, road signs, and road markings in the image data. Claim 8 A method for determining whether image data is within a predetermined range configured for analysis by image analysis software, comprising: receiving said image data from a camera using an electronic processor; using said image analysis software to generate a prediction regarding said image data and a confidence value associated with said prediction regarding said image data; and a cost function of said image analysis software used to generate said prediction regarding said image data. A step of determining the sign of the slope of and calculating perturbation values ​​by multiplying the determined sign by a weight for setting the magnitude of the perturbation values, wherein x is input image data, y is possible classifications for the image data, and θ is the value of the weights included in the image analysis software used to generate the prediction; a step of perturbing the image data using the calculated perturbation values; a step of generating a prediction regarding the perturbed image data and a confidence value associated with the prediction regarding the perturbed image data using the image analysis software; and a step of comparing the confidence value associated with the prediction regarding the perturbed image data with the confidence value associated with the prediction regarding the image data using the electronic processor. A method for determining whether image data is within a predetermined range configured for analysis by image analysis software, comprising the step of determining that the image data received from the camera is within the predetermined range when the difference between the confidence value associated with the prediction regarding the perturbed image data and the confidence value associated with the prediction regarding the image data is greater than or equal to a predetermined threshold value, wherein the image analysis software includes machine learning software trained to analyze the image data, and the machine learning software includes a convolutional neural network. Claim 9 A method for determining whether image data is within a predetermined range configured for analysis by image analysis software, wherein the method further comprises the step of ignoring image data from the camera when the difference between the confidence value associated with the prediction regarding the perturbed image data and the confidence value associated with the prediction regarding the image data is less than the predetermined threshold value. Claim 10 delete Claim 11 delete Claim 12 In claim 8, the method further comprises the step of using data from alternative sensors to generate predictions for controlling the autonomous functions of a vehicle when the image data is outside the predetermined range, a method for determining whether the image data is within a predetermined range configured for analysis by image analysis software. Claim 13 delete Claim 14 In claim 8, the predictions generated by the image analysis software relate to detecting and classifying objects, road signs, and road markings in the image data, and a method for determining whether the image data is within a predetermined range configured for analysis by the image analysis software.

Images