Systems and methods for determining whether image data is within a predetermined range that image analysis software is configured to analyze

Abstract

A system for determining whether image data is within a predetermined range that image analysis software is configured to analyze. The system includes a camera and an electronic processor. The electronic processor is configured to receive image data from the camera and generate a prediction about the image data and a confidence value associated with the prediction. The electronic processor is also configured to perturb the image data using a perturbation value and generate a prediction about the perturbed image data and a confidence value associated with the prediction. The electronic processor is further configured to compare the confidence values ​​and disable an autonomous function of the vehicle when the difference between the confidence value associated with the prediction about the image data and the confidence value associated with the prediction about the image data is less than a predetermined threshold.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 786,590, filed on December 31, 2018, the entire contents of which are hereby incorporated by reference herein. Summary of the invention

[0003] Image sensors are important components in autonomous and partially autonomous driving systems. However, in some cases, image data from image sensors may be analyzed incorrectly. For example, in situations such as severe weather conditions (e.g., rain), weak light, and direct light from the sun, image analysis software may incorrectly generate predictions about the presence of objects in the image, the location of objects in the image, the presence of road markings in the image, the location of road markings in the image, or a combination of the above. The reason why the image analysis software makes incorrect determinations in some cases is because the image analysis software is not configured to analyze the image data obtained in these situations. Therefore, in some embodiments, when image data received from a camera is obtained when the image analysis software is not configured to analyze, it is desirable to ignore the image data. In some embodiments, when image data received from a camera is obtained when the image analysis software is not configured to analyze, it is desirable to disable at least some autonomous functions of the vehicle. The embodiments herein describe a method and system for detecting whether received image data is obtained when the image analysis software is not configured to analyze.

[0004] One embodiment provides a system for determining whether image data is within a predetermined range that image analysis software is configured to analyze. The system includes a camera and an electronic processor. The electronic processor is configured to receive image data from the camera and generate a prediction about the image data and a confidence value associated with the prediction about the image data using the image analysis software. The electronic processor is also configured to perturb the image data using a perturbation value and generate a prediction about the perturbed image data and a confidence value associated with the prediction about the perturbed image data using the image analysis software. The electronic processor is further configured to compare the confidence value associated with the prediction about the perturbed image data with the confidence value associated with the prediction about the image data, and when the difference between the confidence value associated with the prediction about the image data and the confidence value associated with the prediction about the image data is less than a predetermined threshold, disable the autonomous function of the vehicle.

[0005] Another embodiment provides a method for determining whether image data is within a predetermined range that image analysis software is configured to analyze. The method includes: receiving image data from a camera using an electronic processor; and using the image analysis software, generating a prediction about the image data and a confidence value associated with the prediction about the image data. The method also includes: perturbing the image data using a perturbation value, and using the image analysis software, generating a prediction about the perturbed image data and a confidence value associated with the prediction about the perturbed image data. The method further includes: comparing, using the electronic processor, the confidence value associated with the prediction about the perturbed image data and the confidence value associated with the prediction about the image data. The method also includes: when the difference between the confidence value associated with the prediction about the image data and the confidence value associated with the prediction about the image data is less than a predetermined threshold, using data from an alternative sensor to generate a prediction to control an autonomous function of the vehicle.

[0006] Other aspects, features, and embodiments will become apparent by consideration of the detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram of a system for determining whether image data is within a predetermined range that image analysis software is configured to analyze, according to some embodiments.

[0008] Figure 2 According to some embodiments Figure 1 Block diagram of the electronic controller of the system.

[0009] Figure 3 is used according to some embodiments to determine whether image data is within a predetermined range that the image analysis software is configured to analyze Figure 1 Flow chart of a method of a system. DETAILED DESCRIPTION

[0010] Before explaining any embodiments in detail, it is to be understood that the present disclosure is not intended to be limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. The embodiments are capable of other configurations and of being practiced or carried out in various ways.

[0011] A plurality of hardware- and software-based devices, as well as a plurality of differently structured components, may be used to implement various embodiments. In addition, embodiments may include hardware, software, and electronic components or modules, which, for purposes of discussion, may be illustrated and described as if most components were implemented in hardware only. However, a person of ordinary skill in the art will appreciate, based on a reading of this detailed description, that in at least one embodiment, the electronic-based aspects of the present invention may be implemented in software executable by one or more processors (e.g., software stored on a non-transitory computer-readable medium). For example, the "control unit" and "controller" described in the specification may include one or more electronic processors, one or more memory modules including a non-transitory computer-readable medium, one or more input / output interfaces, one or more application-specific integrated circuits (ASICs), and various connections (e.g., a system bus) connecting the various components.

[0012] Figure 1 A system 100 for determining whether image data is within a predetermined range that image analysis software is configured to analyze is illustrated. The system 100 includes a vehicle 105. Although illustrated as a four-wheeled vehicle, the vehicle 105 may encompass vehicles of various types and designs. For example, the vehicle 105 may be a car, a motorcycle, a truck, a bus, a semi-tractor, and the like. In the illustrated example, the vehicle 105 includes an electronic controller 110, a camera 115, a steering system 120, an accelerator 125, and a brake 130. The components of the vehicle 105 may have various configurations, and various communication types and protocols may be used.

[0013] The electronic controller 110 may be communicatively coupled to the camera 115, steering system 120, accelerator 125, and brake 130 via various wired or wireless connections. For example, in some embodiments, the electronic controller 110 is directly coupled to each of the above-listed components of the vehicle 105 via dedicated wires. In other embodiments, the electronic controller 110 is communicatively coupled to one or more components via a shared communication link such as a vehicle communication bus (e.g., a controller area network (CAN) bus) or a wireless connection.

[0014] Each component of the vehicle 105 may communicate with the electronic controller 110 using a variety of communication protocols. Figure 1 The embodiment illustrated in FIG. 1 only provides one example of the components and connections of the vehicle 105. However, these components and connections may be configured in other ways than those illustrated and described herein. For example, it should be understood that the electronic controller 110 may include one or more cameras instead of Figure 1 A single camera 115 is illustrated in FIG. 1 , and the cameras included in the vehicle 105 may be installed at different locations on the interior and exterior of the vehicle 105 .

[0015] Figure 2 1 is a block diagram of an electronic controller 110 of a vehicle 105. The electronic controller 110 includes a plurality of electrical and electronic components that provide power, operational control, and protection to components and modules within the electronic controller 110. The electronic controller 110 includes, among other things, an electronic processor 200 (such as a programmable electronic microprocessor, microcontroller, or similar device), a memory 205 (e.g., a non-transitory machine-readable memory), and an input / output interface 210. The electronic processor 200 is communicatively connected to the memory 205 and the input / output interface 210. The electronic processor 200 cooperates with the memory 205 and the input / output interface 210 and is configured to, among other things, implement the methods described herein.

[0016] As will be described in further detail below, the memory 205 includes computer executable instructions (or software) for, among other things, determining whether the image data is within a predetermined range (and in some cases, obtained) that the image analysis software is trained to analyze. Figure 2 In the example illustrated in , the memory 205 includes: image analysis software 215, which includes a convolutional neural network (CNN) 220 (or more broadly machine learning software) and autonomous function software 225. The CNN 220 is trained to make predictions related to detecting and classifying objects, road markings, road signs, etc. in image data received from the camera 115. The autonomous function software 225 relies on the predictions made by the image analysis software 215 (e.g., CNN 220) using the image data from the camera 115 to provide autonomous functions to the vehicle 105 by controlling the steering system 120, accelerator 125, and brake 130, among other things. The autonomous function software 225 can be, for example, automatic cruise control (ACC), an automatic braking system, an automatic parking system, etc. It should be understood that the memory 205 may include more than 100 pieces of information. Figure 2 The image analysis software 215 may include more, fewer, or different software components than those illustrated in the figure. Although described herein as including a CNN, it should be noted that the image analysis software 215 may include different types of machine learning software, such as a decision tree or a Bayesian network. In some embodiments, the image analysis software 215 may not include machine learning software.

[0017] The electronic controller 110 may be implemented in several independent controllers (e.g., programmable electronic controllers), each of which is configured to perform a specific function or sub-function. Additionally, the electronic controller 110 may include sub-modules that include additional electronic processors, memories, or application-specific integrated circuits (ASICs) for handling input / output functions, signal processing, and application of the methods listed below. In other embodiments, the electronic controller 110 includes additional, fewer, or different components.

[0018] Figure 3 An example of a method 300 for determining whether image data is within a predetermined range that the image analysis software is configured to analyze is illustrated. The method 300 is performed by an electronic processor 200 executing the image analysis software 215. It should be understood that although the example method 300 is described below in terms of a CNN trained to analyze image data, the method 300 is more generally applicable to image analysis software configured to analyze image data. At step 305, the electronic processor 200 receives image data from the camera 115. At step 310, the electronic processor 200 calculates a perturbation value specific to the CNN 220 and the received image data. A perturbation value is a value that, when added to the image data, alters the image data, effectively creating noise in the image data. The electronic processor 200 calculates the perturbation value using the gradient of a cost function of the CNN 220. In some embodiments, the cost function is defined as ,in x is the input image data, y are possible classifications of the image data, and is a weight value included in CNN 220. The perturbation value is determined to determine the cost function The sign of the gradient (positive or negative) is multiplied by the weight At step 315, the image data is perturbed using the perturbation value (the perturbation value is added to each pixel of the image data). At step 320, the electronic processor 200 executes the CNN 220 to generate a prediction for the perturbed image data (classify the perturbed image data into one of a plurality of categories). For example, the CNN 220 may determine whether the lane marking to the right of the vehicle 105 is a solid line or a dashed line. When the electronic processor 200 generates a prediction, the prediction is associated with a confidence value. The confidence value indicates the likelihood that the prediction is correct.

[0019] At step 325, the electronic processor 200 also executes the CNN 220 to generate a prediction about the image data (classify the image data into one of a plurality of categories), and determine a confidence value associated with the prediction about the image data. It should be noted that although step 325 is Figure 3 3. In the example shown in FIG. 3, steps 310-320 are shown as being performed in parallel, but in some embodiments, these steps may be performed sequentially.

[0020] At step 330, the electronic processor 200 compares the confidence value associated with the prediction made based on the disturbed image data with the confidence value associated with the prediction made based on the unperturbed image data. At step 335, when the difference between the confidence values ​​is less than a predetermined threshold, the electronic processor 200 determines that the image data received from the camera 115 is outside the predetermined range that the CNN 220 is trained to analyze. When the image data received from the camera 115 is outside the predetermined range that the CNN 220 is trained to analyze, the electronic processor 200 determines that the prediction made by the CNN 220 is unreliable. In some embodiments, when the image data received from the camera 115 is outside the predetermined range that the CNN 220 is trained to analyze, and the autonomous function software 225 relies on the prediction generated by the CNN 220, the electronic processor 200 disables the autonomous function of the vehicle 105 controlled by the autonomous function software 225.

[0021] In other embodiments, instead of disabling the autonomous function software 225, the electronic processor 200 ignores (disregards) the image data from the camera 115 when the image data received from the camera 115 is outside of the predetermined range that the CNN 220 is trained to analyze. In the event that the electronic processor 200 ignores the image data from the camera 115, the electronic processor 200 uses data from an alternative sensor to generate predictions using the CNN 220 (or other types of software) and perform the autonomous functions of the autonomous function software 225. Examples of data from alternative sensors include: image data from other cameras, or data from sensors such as radar sensors, LIDAR sensors, ultrasonic sensors, etc.

[0022] It should be understood that the image analysis software 215 may include more than one CNN, and each CNN may be trained to detect different content in the received image data. For example, one CNN may be trained to detect road markings (e.g., lane markings), while another CNN is trained to detect objects such as people, animals, and vehicles. In some embodiments, when the image data is determined to be outside the predetermined range that the CNN is trained to analyze, the autonomous function controlled by the autonomous function software 225 is disabled, regardless of whether the autonomous function software 225 relies on the predictions made by the CNN based on the image data.

[0023] In the foregoing description, specific embodiments have been described. However, it will be appreciated by those skilled in the art that various modifications and changes may be made without departing from the scope of the invention as set forth in the appended claims. Therefore, the description and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this teaching.

[0024] In this document, relational terms such as first and second, top and bottom, etc., may be used alone to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other variation thereof are intended to cover non-exclusive inclusions, such as a process, method, article, or apparatus that includes, has, includes, or contains a list of elements that includes not only those elements, but may also include other elements that are not explicitly listed or are inherent to such process, method, article, or apparatus. Elements beginning with "includes ... one," "have ... one," "includes ... one," or "contains ... one" do not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes, has, includes, or contains the element without more constraints. Unless otherwise expressly stated herein, the terms "a" and "an" are defined as one or more. The terms "substantially," "substantially," "approximately," "about," or any other versions thereof, are defined as approximating what one of ordinary skill in the art understands, and in one non-limiting embodiment, within 10%, in another embodiment within 5%, in another embodiment within 1%, and in another embodiment within 0.5%. The term "coupled," as used herein, is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

[0025] Various features, advantages and embodiments are set forth in the following claims.

Claims

1. A system for determining whether image data is within a predetermined range that image analysis software is configured to analyze, the system comprising: camera; as well as An electronic processor configured to: Receive image data from the camera; Using image analysis software, generating predictions about the image data and confidence values ​​associated with the predictions about the image data; perturbing the image data using a perturbation value; Using image analysis software, generating predictions regarding the perturbed image data and confidence values ​​associated with the predictions regarding the perturbed image data; comparing a confidence value associated with a prediction regarding the perturbed image data and a confidence value associated with a prediction regarding the image data; as well as When a difference between a confidence value associated with a prediction regarding the image data and a confidence value associated with the prediction regarding the image data is less than a predetermined threshold, determining that the image data received from the camera is outside of a predetermined range that the image analysis software is configured to analyze and disabling autonomous functionality of the vehicle.

2. The system according to claim 1, wherein: The disabled autonomous functions rely on predictions about the image data.

3. The system according to claim 1, wherein: The image analysis software includes machine learning software trained to analyze image data.

4. The system according to claim 3, wherein: The machine learning software includes a convolutional neural network.

5. The system according to claim 4, wherein: The electronic processor is configured to calculate a disturbance value.

6. The system according to claim 5, wherein: The electronic processor is configured to calculate the disturbance value by: Determine the sign of the gradient of the cost function of the convolutional neural network; and The weight is multiplied by the sign.

7. The system according to claim 1, wherein: The predictions generated by the image analysis software involve detecting and classifying objects, road signs, and road markings in the image data.

8. A method of determining whether image data is within a predetermined range that image analysis software is configured to analyze, the method comprising: receiving image data from a camera using an electronic processor; Using image analysis software, generating predictions about the image data and confidence values ​​associated with the predictions about the image data; perturbing the image data using a perturbation value; Using image analysis software, generating predictions regarding the perturbed image data and confidence values ​​associated with the predictions regarding the perturbed image data; comparing, with an electronic processor, a confidence value associated with a prediction regarding the perturbed image data and a confidence value associated with a prediction regarding the image data; as well as When a difference between a confidence value associated with a prediction about the image data and a confidence value associated with the prediction about the image data is less than a predetermined threshold, determining that the image data received from the camera is outside of a predetermined range that the image analysis software is configured to analyze and using data from an alternative sensor to generate a prediction to control an autonomous function of the vehicle.

9. The method according to claim 8, further comprising: When a difference between a confidence value associated with a prediction regarding the image data and a confidence value associated with the prediction regarding the image data is less than a predetermined threshold, the image data from the camera is ignored.

10. The method according to claim 8, wherein: The image analysis software includes machine learning software trained to analyze image data.

11. The method according to claim 10, wherein: The machine learning software includes a convolutional neural network.

12. The method of claim 11, further comprising calculating a disturbance value.

13. The method according to claim 12, wherein: Calculating the perturbation value includes: determining the sign of the gradient of the cost function of the convolutional neural network; and The weight is multiplied by the sign.

14. The method according to claim 8, wherein: The predictions generated by the image analysis software involve detecting and classifying objects, road signs, and road markings in the image data.

Images