Abnormality detection device and method for object recognition, and storage medium

Abstract

An image processing ECU as an abnormality detection device for object recognition detects abnormalities in the recognition of surrounding objects of a vehicle using a distance measuring sensor (10) and an external camera (20). The distance measuring sensor is mounted on the vehicle and generates a background light image in addition to a reflected light image containing distance information. The external camera is mounted on the vehicle and generates a camera image. The image processing ECU comprises: a speed state grasping unit for acquiring the speed state of the vehicle; an image information acquiring unit for acquiring the background light image and the camera image; and a projection error detecting unit for detecting the projection error between the background light image and the camera image when the vehicle generates the background light image and the camera image in a speed state below a low speed area.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on patent application No. 2019-164861 filed in Japan on September 10, 2019, and the contents of the basic application are cited by reference in their entirety. Technical Field

[0003] The disclosure based on this specification relates to the detection of abnormality in the recognition of surrounding objects of a vehicle. Background Art

[0004] The device disclosed in Patent Document 1 recognizes objects around a vehicle using the measurement results of a distance measuring sensor and an external camera. The device determines the reliability of the object detection result and whether the detection state is stable based on the state of image loss of the object reflected in the image.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-97765

[0006] The distance measuring sensor and the external camera are mounted on vehicles that move to various environments. Therefore, sometimes an abnormality occurs in the recognition of surrounding objects caused by the distance measuring sensor and the external camera themselves. However, the device of Patent Document 1 cannot detect such an abnormality. Summary of the invention

[0007] One of the purposes of the disclosure of this specification is to provide an abnormality detection device for object recognition, an abnormality detection program for object recognition, and an abnormality detection method for object recognition, which can detect abnormalities in peripheral object recognition of a vehicle using a ranging sensor and an external camera with high accuracy.

[0008] One method disclosed herein provides an abnormality detection device for object recognition, which detects abnormalities in the recognition of surrounding objects of a vehicle using a distance measuring sensor and an external camera, wherein the distance measuring sensor is mounted on the vehicle, and generates a reflected light image containing distance information by sensing reflected light reflected from an object by light irradiation through a light receiving element, and generates a background light image having the same coordinate system as the reflected light image by sensing background light relative to the reflected light through the light receiving element, and the external camera is mounted on the vehicle, and generates a camera image by detecting incident light from the outside through a camera element, wherein the abnormality detection device for object recognition comprises:

[0009] A speed state control unit controls the speed state of the vehicle;

[0010] An image information acquisition unit, which acquires a background light image and a camera image; and

[0011] The projection error detection unit detects a projection error between the background light image and the camera image when the background light image and the camera image are generated in a state where the vehicle has a speed lower than a predetermined speed range.

[0012] One method disclosed herein provides an object recognition anomaly detection program for detecting anomalies in the recognition of surrounding objects of a vehicle, wherein:

[0013] The object recognition anomaly detection program causes at least one processing unit to execute the following processing:

[0014] A process of obtaining a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor mounted on a vehicle, wherein a reflected light image containing distance information is generated by sensing reflected light reflected from an object by light irradiation through a light receiving element, and a background light image having the same coordinate system as the reflected light image is generated by sensing background light relative to the reflected light through the light receiving element;

[0015] A process of acquiring a camera image, wherein the camera image is an image generated by an external camera mounted on the vehicle, and the camera image is generated by detecting incident light from the outside world by a camera element; and

[0016] A process for detecting a projection error between a background light image and a camera image when a background light image and a camera image are generated in a state where the vehicle has a speed lower than a predetermined speed range.

[0017] One method disclosed herein provides an abnormality detection method for object recognition, which detects abnormalities in the recognition of surrounding objects of a vehicle, wherein:

[0018] The processing performed by at least one processing unit includes:

[0019] Acquire a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor mounted on the vehicle, wherein the reflected light image containing distance information is generated by sensing reflected light reflected from an object by light irradiation through a light receiving element, and the background light image having the same coordinate system as the reflected light image is generated by sensing background light relative to the reflected light through the light receiving element;

[0020] Acquiring a camera image, wherein the camera image is an image generated by an external camera mounted on the vehicle, and the camera image is generated by detecting incident light from the outside world by a camera element; and

[0021] When a background light image and a camera image are generated in a state where the vehicle has a speed lower than a predetermined speed range, a projection error between the background light image and the camera image is detected.

[0022] According to these methods, the projection error between the background light image and the camera image is detected when the vehicle is at a speed less than the specified speed range. In such a speed state, the reduction in detection accuracy caused by the deviation between the measurement timing of the background light image generated by the ranging sensor and the measurement timing of the camera image generated by the external camera can be suppressed. For example, the deviation in the projection position of the object between the background light image and the camera image caused by the movement of the vehicle during the deviation in the measurement timing is reduced, so even if the position deviation is included in the projection error, the position deviation has little effect on the increase of the projection error. Therefore, it is possible to detect abnormalities in the recognition of peripheral objects of the vehicle using the ranging sensor and the external camera with high accuracy.

[0023] In addition, the reference numerals in parentheses in the claims and the like exemplarily indicate the correspondence relationship with the parts of the embodiments described later, and are not intended to limit the technical scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a diagram showing an overall image of the surrounding monitoring image information system and the driving assistance ECU according to the first embodiment.

[0025] Figure 2 It is a diagram showing a state where the distance measuring sensor and the outside world camera according to the first embodiment are mounted on a vehicle.

[0026] Figure 3 1 is a block diagram showing the structure of the image processing ECU according to the first embodiment.

[0027] Figure 4 It is a diagram showing corresponding points on the background light image side according to the first embodiment.

[0028] Figure 5 It is a diagram showing the relationship among the distance measuring sensor, the external camera, and the three-dimensional coordinates according to the first embodiment.

[0029] Figure 6 It is a diagram showing corresponding points, projection positions, and projection errors on the camera image side according to the first embodiment.

[0030] Figure 7 This is a graph for explaining the increase in projection error in the first embodiment.

[0031] Figure 8 This is a flowchart of the image processing ECU according to the first embodiment. DETAILED DESCRIPTION

[0032] An embodiment will be described based on the drawings.

[0033] (First Embodiment)

[0034] like Figure 1As shown, the object recognition abnormality detection device of the first embodiment of the present invention is used for the recognition of the surrounding objects of the vehicle 1, and is constituted as an image processing ECU (Electronic Control Unit: electronic control unit) 30 mounted on the vehicle 1. The image processing ECU 30, together with the distance measuring sensor 10 and the outside camera 20, constitutes a surrounding monitoring image information providing system 100. The surrounding monitoring image information providing system 100 of this embodiment can provide the surrounding monitoring image information based on the measurement results of the distance measuring sensor 10 and the outside camera 20 to the driving assistance ECU 50 and the like.

[0035] The image processing ECU 30 is communicatively connected to a communication bus of an in-vehicle network mounted on the vehicle 1. The image processing ECU 30 is one of a plurality of nodes provided in the in-vehicle network. In addition to the distance measuring sensor 10 and the external camera 20, the driving assistance ECU 50 and the like are connected to the in-vehicle network as nodes.

[0036] The driving assistance ECU 50 is a structure that includes a computer having a processor, a RAM (Random Access Memory), a storage unit, an input / output interface, and a bus connecting them as a main body. The driving assistance ECU 50 has at least one of a driving assistance function that assists the driver's driving operation in the vehicle 1 and a driving proxy function that can perform the driver's driving operation. The driving assistance ECU 50 recognizes the surrounding objects of the vehicle 1 based on the surrounding monitoring image information provided by the surrounding monitoring image information providing system 100 by executing the object recognition program stored in the storage unit by the processor. The driving assistance ECU 50 realizes the automatic driving or advanced driving assistance of the vehicle 1 corresponding to the recognition result by executing the program stored in the storage unit by the processor.

[0037] The driving assistance ECU 50 implements the recognition of surrounding objects using, for example, semantic segmentation by executing the object recognition program. In the storage unit of the driving assistance ECU 50, as a component of the object recognition program, an object recognition model 51 based on a neural network is constructed (see also Figure 3 ). The neural network can adopt a structure called SegNet that combines an encoder and a decoder, for example. In addition, surrounding object recognition can also be object recognition using a bounding box.

[0038] Next, the details of the distance measuring sensor 10 , the outside camera 20 , and the image processing ECU 30 included in the surrounding monitoring image information providing system 100 will be described in order.

[0039] The distance measuring sensor 10 is, for example, a SPAD LiDAR (Single Photon Avalanche Diode Light Detection And Ranging) disposed in front of the vehicle 1 or on the roof of the vehicle 1. The distance measuring sensor 10 can measure at least a measurement range MA1 in the front of the periphery of the vehicle 1.

[0040] The distance measuring sensor 10 includes a light emitting unit 11, a light receiving unit 12, a control unit 13, etc. The light emitting unit 11 scans the light source by using a movable optical component (eg, a polygon mirror). Figure 2 The measurement range MA1 shown is irradiated with a light beam emitted from a light source. The light source is, for example, a semiconductor laser (Laser diode), and emits a near-infrared light beam that cannot be visually recognized by passengers (drivers, etc.) and outsiders according to an electrical signal from the control unit 13 .

[0041] The light receiving unit 12 collects reflected light of the irradiated light beam reflected by an object within the measurement range MA1 or background light relative to the reflected light using, for example, a condenser lens, and makes the reflected light incident on the light receiving element 12 a .

[0042] The light receiving element 12a is an element that converts light into an electrical signal by photoelectric conversion, and is a SPAD light receiving element that achieves high sensitivity by amplifying the detection voltage. In the light receiving element 12a, for example, in order to detect reflected light in the near-infrared region, a CMOS sensor is used that sets the sensitivity of the near-infrared region higher than that of the visible region. The sensitivity can also be adjusted by providing an optical filter in the light receiving portion 12. The light receiving element 12a has a plurality of light receiving pixels arranged in a one-dimensional direction or a two-dimensional direction in an array.

[0043] The control unit 13 is a unit for controlling the light emitting unit 11 and the light receiving unit 12. The control unit 13 is, for example, arranged on a substrate shared with the light receiving element 12a, and is mainly composed of a processor in a broad sense such as a microcomputer or FPGA (Field-Programmable Gate Array). The control unit 13 realizes a scanning control function, a reflected light measurement function, and a background light measurement function.

[0044] The scanning control function is a function of controlling light beam scanning. The control unit 13 causes the light beam to pulse a plurality of times from the light source at the timing based on the operation clock of the clock oscillator provided in the distance measuring sensor 10 and operates the movable optical component.

[0045] The reflected light measurement function is a function for measuring the intensity of the reflected light by reading the voltage value based on the reflected light received by each light-receiving pixel, for example, using a rolling shutter method, in accordance with the timing of the light beam scanning. In the measurement of the reflected light, the distance from the distance measuring sensor 10 to the object that reflected the reflected light can be measured by detecting the time difference between the timing of the light beam emission and the timing of the light reception of the reflected light. By measuring the reflected light, the control unit 13 can generate a reflected light image ImR, which is image-like data that associates the intensity of the reflected light and the distance information of the object that reflected the reflected light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1.

[0046] The background light measurement function is a function of reading out the voltage value based on the background light received by each light-receiving pixel at a timing immediately before measuring the reflected light, and measuring the intensity of the background light. Here, the background light refers to the incident light that does not substantially include the reflected light and enters the light-receiving element 12a from the measurement range MA1 in the outside world. The incident light includes natural light and display light emitted from a display in the outside world. By measuring the background light, the control unit 13 can generate a background light image ImL, which is image-like data that associates the intensity of the background light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1.

[0047] The reflected light image ImR and the background light image ImL are perceived by a common light receiving element 12a and acquired from a common optical system including the light receiving element 12a. Therefore, the coordinate system of the reflected light image ImR and the coordinate system of the background light image ImL can be regarded as the same coordinate system that is consistent with each other. Moreover, it can be said that there is almost no deviation in the measurement timing between the reflected light image ImR and the background light image ImL (for example, less than 1ns). Therefore, the reflected light image ImR and the background light image ImL can be regarded as also acquiring synchronization.

[0048] For example, in the present embodiment, image data storing three channels of data of intensity of reflected light, distance to an object, and intensity of background light corresponding to each pixel is sequentially output to the image processing ECU 30 as a sensor image.

[0049] The external camera 20 is, for example, a camera disposed inside the front windshield of the vehicle 1. The external camera 20 can measure at least a forward measurement range MA2 of the external environment of the vehicle 1, more specifically, a measurement range MA2 at least partially overlapping the measurement range MA1 of the distance measuring sensor 10.

[0050] The external camera 20 includes a light receiving unit 22 and a control unit 23. The light receiving unit 22 collects incident light (background light) entering from a measurement range MA2 outside the camera using, for example, a light receiving lens, and makes the light enter a camera element 22a.

[0051] The camera element 22a is an element that converts light into an electrical signal by photoelectric conversion, and can be, for example, a CCD sensor or a CMOS sensor. In the camera element 22a, in order to efficiently receive natural light in the visible area, etc., the sensitivity of the visible area is set high relative to the near-infrared area. The camera element 22a has a plurality of light-receiving pixels (equivalent to so-called sub-pixels) arranged in an array in a two-dimensional direction. For example, red, green, and blue color filters are configured for mutually adjacent light-receiving pixels. Each light-receiving pixel receives visible light of a color corresponding to the configured color filter. By measuring the intensity of red, green, and blue, respectively, the camera image ImC taken by the external camera 20 is an image with a higher resolution than the reflected light image ImR and the background light image ImL, and can be a color image of the visible area.

[0052] The control unit 23 is a unit that controls the light receiving unit 22. The control unit 23 is disposed on a substrate shared with the camera element 22a, for example, and is mainly composed of a microcomputer or a processor in a broad sense such as an FPGA. The control unit 23 realizes a photographing function.

[0053] The shooting function is a function of shooting the color image described above. The control unit 23 reads out the voltage value based on the incident light received by each light-receiving pixel, for example, using a global shutter method, at the timing based on the action clock of the clock oscillator provided in the external camera 20, and senses and measures the intensity of the incident light. The control unit 23 can generate a camera image ImC, which is image-like data that associates the intensity of the incident light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA2. Such camera images ImC are sequentially output to the image processing ECU 30.

[0054] The image processing ECU 30 is an electronic control device that performs composite processing on the surrounding monitoring image information based on the measurement results of the distance measuring sensor 10 and the external camera 20. Figure 1As shown, the image processing ECU 30 is a structure that includes a computer having a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, and a bus connecting them as a main body. The processing unit 31 is hardware for calculation processing combined with the RAM 32. The processing unit 31 is a structure that includes at least one CPU (Central Processing Unit), a GPU (Graphical Processing Unit), a RISC (Reduced Instruction Set Computer) and other calculation cores. The processing unit 31 can also be a structure that also includes an FPGA and an IP core with other dedicated functions. The processing unit 31 performs various processes for realizing the functions of each functional unit described later by accessing the RAM 32. The storage unit 33 is a structure that includes a non-volatile storage medium. Various programs (such as an abnormality detection program for object recognition) executed by the processing unit 31 are stored in the storage unit 33.

[0055] The image processing ECU 30 includes a plurality of functional units by executing a program stored in the storage unit 33 through the processing unit 31. Specifically, Figure 3 As shown, functional units such as an image information acquisition unit 41, an image registration unit 42, a speed state grasping unit 43, a projection error detection unit 44, an abnormality determination unit 45, and an abnormality notification information output unit 46 are constructed.

[0056] In addition, in the image processing ECU 30, the measured position data 47 and the projection error detection data 48 are stored in the storage unit 33. The measured position data 47 includes data related to the positional relationship between the measured position of the distance measuring sensor 10 and the measured position of the external camera 20. The data related to the positional relationship may also be data on the position and orientation of the light receiving element 12a of the distance measuring sensor 10 in the vehicle 1 and the position and orientation of the camera element 22a of the external camera 20. Alternatively, data obtained by transforming these data and indicating the values ​​of the components of the transformation matrix described later may also be data related to the positional relationship.

[0057] The projection error detection data 48 is data recording the detection result of the projection error detection unit 44 and is used for determination by the abnormality determination unit 45. For example, the projection error detection data 48 is data recording a plurality of sets of detection date and time, an image of a detection source, and a value of the detected projection error PE.

[0058] The image information acquisition unit 41 acquires the reflected light image ImR and the background light image ImL from the distance measuring sensor 10, and acquires the camera image ImC from the external camera 20. The image information acquisition unit 41 provides the latest set of the reflected light image ImR, the background light image ImL, and the camera image ImC to the image registration unit 42 and the projection error detection unit 44.

[0059] The image registration unit 42 determines the correspondence between the coordinate systems of the reflected light image ImR and the background light image ImL and the coordinate system of the camera image ImC, i.e., image registration. The image registration unit 42 sequentially provides the registered reflected light image ImR, the background light image ImL, and the camera image ImC to the driving assistance ECU 50.

[0060] The speed state grasping unit 43 grasps the speed state of the vehicle 1. The speed state grasping unit 43 can grasp the speed state of the vehicle 1 by, for example, acquiring the speed state of the vehicle 1 from the speed sensor of the vehicle 1. Alternatively, the speed state grasping unit 43 may not obtain the vehicle speed information from the vehicle speed sensor, but may grasp the speed state of the vehicle 1 by comparing the two most recent background light images ImL (or camera images ImC) and estimating the speed state of the vehicle 1 based on the movement amount of the background etc. reflected in the image.

[0061] like Figures 4 to 6 As shown, the projection error detection unit 44 compares the background light image ImL with the camera image ImC to detect the projection error PE between the background light image ImL and the camera image ImC. The projection error detection unit 44 has a speed state determination function, a corresponding point acquisition function, a corresponding point projection function, and a projection error measurement function.

[0062] The speed state determination function is a function for determining whether the speed state of the vehicle 1 is a condition suitable for detecting the projection error PE, in other words, a function for determining whether to perform the detection process of the projection error PE. With regard to the stopped state of the vehicle 1, since the position accuracy of the stationary object SO measured by the distance sensor 10 and the position accuracy of the stationary object SO measured by the external camera 20 are substantially the highest, it can be said that it is a condition suitable for detecting the projection error PE. On the other hand, in the running state of the vehicle 1, the greater the speed, the greater the deviation between the measurement timing of the distance sensor 10 and the measurement timing of the external camera 20, which will cause a position deviation between the projection position of the stationary object SO in the background light image ImL and the projection position of the stationary object SO in the camera image ImC. Therefore, if the speed of the vehicle 1 increases, the condition suitable for detecting the projection error PE is not met.

[0063] Specifically, the projection error detection unit 44 determines whether the background light image ImL and the camera image ImC, which are detection targets of the projection error PE, are generated in the stopped state of the vehicle 1 based on the speed state grasped by the speed state grasping unit 43. If they are generated in the stopped state, the process for detecting the projection error PE with the background light image ImL and the camera image ImC as the objects is performed. If not, the process for detecting the projection error PE with the background light image ImL and the camera image ImC as the objects is terminated.

[0064] The corresponding point acquisition function is a function of acquiring corresponding points CPa and CPb from the background light image ImL and the camera image ImC that form a group. The corresponding points CPa and CPb are feature points (the same point SaP) of a common stationary object SO that appears in both the background light image ImL and the camera image ImC (see also Figure 4 , 6 ).

[0065] Specifically, when the projection error detection unit 44 determines that the background light image ImL and the camera image ImC are generated in the stopped state, it extracts the stationary object SO that is commonly reflected in the background light image ImL and the camera image ImC. The extracted stationary object SO is preferably a stationary object whose shape is easy to extract, such as a sign, a logo, or a traffic light.

[0066] The projection error detection unit 44 obtains the feature points (e.g., corners) of the extracted stationary object SO as corresponding points CPa and CPb. One or more pairs of corresponding points CPa and CPb are obtained between the background light image ImL and the camera image ImC. In particular, in the present embodiment, when multiple pairs of feature points can be obtained, such as when multiple stationary objects SO are extracted, the projection error detection unit 44 preferably obtains multiple pairs of corresponding points CPa and CPb. In addition, Figure 4 , 6 In order to simplify the diagram, a pair of corresponding points CPa and CPb are recorded.

[0067] The corresponding point projection function is a function of projecting the corresponding point CPa on the background light image side onto the camera image for each pair of acquired corresponding points CPa and CPb. Figure 5 As shown, the projection error detection unit 44 determines the three-dimensional coordinates P (x, y, z) of the corresponding point CPa on the background light image side by referring to the distance information of the reflected light image ImR. That is, the background light image ImL does not contain distance information. Therefore, the corresponding point CPa is two-dimensional coordinate information on the image plane of the background light image ImL. By combining the distance information of the reflected light image ImR with the two-dimensional coordinate information, the corresponding point CPa can be expanded into three-dimensional coordinate information. Figure 4As shown, the determination of the three-dimensional coordinates P (x, y, z) can be achieved by considering the reflected light image ImR and the background light image ImL as the same coordinate system and obtaining synchronization. The three-dimensional coordinates P (x, y, z) are expressed by a coordinate system obtained by adding the z axis along the depth direction perpendicular to the image plane to the two-dimensional coordinates (x, y) of the image plane of the background light image ImL.

[0068] The projection error detection unit 44 projects the three-dimensional coordinates P (x, y, z) to the projection position PP on the image plane of the camera image ImC by using the coordinate transformation of the transformation matrix. Here, the transformation matrix is ​​a matrix that applies a rotation operation and a translation operation to the three-dimensional coordinates P (x, y, z) and applies an operation of inversely projecting the transformed coordinates to the background light image ImL. The transformation matrix can be described in the form of a single matrix or in the form of the product of multiple matrices (for example, the product of the inverse projection matrix, the translation matrix T, and the rotation matrix R).

[0069] The projection error measurement function is a function for measuring the positional offset between the position of the corresponding point CPb on the camera image side and the projection position PP projected from the background light image side by coordinate transformation as the projection error PE for each pair of the acquired corresponding points CPa and CPb. The projection error PE is represented by, for example, the distance between the position of the corresponding point CPb on the camera image side and the projection position PP. Alternatively, the projection error PE may also be represented by an angle error after angle conversion based on the optical system of the light receiving unit 22. The result of the detected projection error PE is stored in the storage unit 33 in the form of a group in which the detection date and time, the image of the detection source, and the value of the detected projection error PE are sequentially added to the projection error detection data 48.

[0070] The abnormality determination unit 45 determines an abnormality in the peripheral object recognition based on the projection error PE detected by the projection error detection unit 44. The abnormality mentioned here includes not only a failure of the peripheral object recognition function, but also a state where the accuracy of the peripheral object recognition is reduced to an unacceptable level, or a state where the possibility of reduction is high. If the projection error PE is a large value, it is estimated that at least one of the transformation matrix has a problem and the distance information measured by the distance measuring sensor 10 has a problem.

[0071] The abnormality determination unit 45 determines the following multiple (for example, three) determination conditions while referring to the projection error detection data 48 by accessing the storage unit 33. The abnormality determination unit 45 determines that there is an abnormality in the peripheral object recognition when one or more of the following multiple determination conditions are satisfied. The abnormality determination unit 45 determines that there is no abnormality in the peripheral object recognition when all of the multiple determination conditions are not satisfied.

[0072] The first condition is a condition for judging when a plurality of pairs of corresponding points CPa and CPb are obtained in the same reflected light image ImR and camera image ImC, and projection errors PE are detected for the plurality of pairs. Specifically, as the first condition, the abnormality judgment unit 45 judges whether a projection error PE greater than a prescribed value A greater than a prescribed number of detections K is detected in the same group of the reflected light image ImR and the camera image ImC. The prescribed value A is a noise removal value for removing noise. By appropriately setting the prescribed value A, for example, noise affecting vehicle vibration caused by the internal combustion engine of the vehicle 1 can be removed. The detection number K is a reliability-ensured detection number for improving the reliability of abnormality judgment, and an integer greater than 2 is used.

[0073] The second condition is a condition for judging the most recently detected N projection errors PE regardless of whether they are the same reflected light image ImR and camera image ImC. Specifically, as the second condition, the abnormality judgment unit 45 judges whether the average value of the most recently detected predetermined multiple times N projection errors PE is greater than a predetermined value B. The number N is, for example, a reliability assurance number for improving the reliability of abnormality judgment, and an integer greater than 2 is used. The predetermined value B is set based on the recognition performance of the object recognition performed by the driving assistance ECU 50, and is the value of the allowable projection error allowed in maintaining the accuracy of the object recognition. The predetermined value B may be the same value as the predetermined value A, or a different value.

[0074] The third condition is a condition determined based on the long-term change of the projection error PE. As the third condition, the abnormality determination unit 45 determines whether the projection error PE continues to increase. Figure 7 As shown, the abnormality determination unit 45 classifies the projection error PE for each period (e.g., each month) among the plurality of periods DP based on the detection date and time of the projection error detection data 48. The abnormality determination unit 45 compares the projection errors PE of each period DP. Figure 7 When the projection error PE increases in the latest three periods DP compared with the previous periods, such as the first month 1M, the second month 2M, and the third month 3M shown, the abnormality determination unit 45 determines that the projection error PE continues to increase.

[0075] When the abnormality determination unit 45 determines that there is an abnormality in the peripheral object recognition, the abnormality notification information output unit 46 generates abnormality notification information for notifying the abnormality and outputs it to an output destination. The output destination is, for example, an information prompting device (e.g., a meter, a HUD, a speaker) of the vehicle 1. By notifying the occupant of the abnormality through the information prompting device, the occupant can implement various responses.

[0076] In addition, the output destination may also be an information collection center outside the vehicle that utilizes communication based on a DCM (Data Communication Module). DCM is a communication module mounted on vehicle 1. DCM sends and receives radio waves between vehicle 1 and surrounding base stations through wireless communications in accordance with communication standards such as LTE (Long Term Evolution) and 5G. In the case where vehicle 1 is a commercial vehicle such as a passenger taxi, a passenger bus, a rental car, a shared vehicle, or an unmanned transport vehicle, an abnormality may also be notified to an information collection center located at a business office.

[0077] Next, use Figure 8 The flowchart of FIG. 1 illustrates the details of an anomaly detection method for detecting anomalies in surrounding object recognition based on an anomaly detection program.

[0078] First, in S11, the projection error detection unit 44 determines whether the background light image ImL and the camera image ImC are generated in the stopped state of the vehicle 1. If an affirmative determination is made in S11, the process proceeds to S12. If a negative determination is made in S11, a series of processes are terminated.

[0079] In S12, the projection error detection unit 44 extracts the stationary object SO that appears in both the background light image ImL and the camera image ImC. After the processing of S12, the process proceeds to S13.

[0080] In S13, the projection error detection unit 44 acquires the corresponding points CPa and CPb of the stationary object SO extracted in S12. After the processing in S13, the process proceeds to S14.

[0081] In S14, the projection error detection unit 44 projects the corresponding point CPa on the background light image side to the projection position PP on the camera image. After the processing of S14, the process proceeds to S15.

[0082] In S15, the projection error detection unit 44 measures the projection error PE based on the projection position PP projected in S14 and the corresponding point CPb on the camera image side. The detection result of the projection error PE is recorded as the projection error detection data 48. After the processing of S15, the process moves to S16.

[0083] In S16, the abnormality determination unit 45 determines an abnormality in the peripheral object recognition by referring to the projection error detection data 48. If it is determined in S16 that an abnormality exists, the process proceeds to S17. If it is determined in S16 that no abnormality exists, the series of processes ends.

[0084] In S17 , the abnormality notification information output unit 46 outputs the abnormality notification information to the output destination. A series of processes are terminated in S17 . In this way, the image processing ECU 30 can detect abnormality in the peripheral object recognition of the vehicle 1 .

[0085] (Effect)

[0086] The effects of the first embodiment described above will be described again below.

[0087] According to the first embodiment, the projection error PE between the background light image ImL and the camera image ImC is detected in a speed state where the vehicle 1 is at a speed below the low speed area. In such a speed state, it is possible to suppress the reduction in detection accuracy caused by the deviation in the measurement timing of the background light image ImL generated by the distance measuring sensor 10 and the measurement timing of the camera image ImC generated by the external camera 20. For example, the deviation in the projection position of the object between the background light image ImL and the camera image ImC caused by the movement of the vehicle 1 during the deviation in the measurement timing is reduced, so even if the position deviation is included in the projection error PE, the position deviation has little effect on the increase of the projection error PE. Therefore, it is possible to detect abnormalities in the recognition of peripheral objects of the vehicle 1 using the distance measuring sensor 10 and the external camera 20 with high accuracy.

[0088] In addition, according to the first embodiment, the projection error PE between the background light image ImL and the camera image ImC is detected when the vehicle 1 is stopped. In the stopped state, the deviation of the projection position of the object between the background light image ImL and the camera image ImC caused only by the movement of the vehicle 1 during the deviation of the measurement timing is eliminated. Therefore, such position deviation is suppressed from being included in the projection error PE. Therefore, it is possible to detect abnormalities in the recognition of the surrounding objects of the vehicle 1 using the distance measuring sensor 10 and the external camera 20 with high accuracy.

[0089] In addition, according to the first embodiment, the distance information is used to determine the three-dimensional coordinates P (x, y, z) of the corresponding point CPa on the background light image side among the corresponding points CPa and CPb corresponding to each other between the background light image ImL and the camera image ImC. The projection source of the projection position PP projected onto the image plane of the camera image ImC is the three-dimensional coordinates P (x, y, z), thereby significantly improving the position accuracy of the projection position PP. Moreover, by measuring the deviation between the position of the corresponding point CPb on the camera image side and the projection position PP on the image plane of the camera image ImC, the projection error PE can be easily detected.

[0090] According to the first embodiment, abnormality in peripheral object recognition is determined based on the detected projection error PE. By using the projection error PE detected with high accuracy, the accuracy of abnormality determination is also high.

[0091] In addition, according to the first embodiment, in the same group of the reflected light image ImR and the camera image ImC, when a projection error PE greater than a specified value A greater than a specified number of detections K is detected, it is determined that there is an abnormality in the peripheral object recognition. By considering multiple projection errors PE in the same group, the reliability of abnormality judgment can be greatly improved.

[0092] In addition, according to the first embodiment, when the average value of the projection errors PE detected recently for a predetermined number of times N is greater than a predetermined value B, it is determined that there is an abnormality in the peripheral object recognition. By considering the average value of the projection errors PE detected recently for a predetermined number of times, the reliability of the abnormality determination can be greatly improved.

[0093] According to the first embodiment, when the projection error PE continues to increase, it is determined that there is an abnormality in the peripheral object recognition. By checking the continuous increase of the projection error PE, it is possible to take preventive measures such as amplifying the projection error PE predicted in the future to an unacceptable range.

[0094] (Other Embodiments)

[0095] Although one embodiment has been described above, the present invention should not be construed as being limited to this embodiment, and can be applied to various embodiments without departing from the gist of the present invention.

[0096] Specifically, as a modification example 1, the projection error detection unit 44 can also determine whether the background light image ImL and the camera image ImC as the detection object of the projection error PE are generated in a speed state of a speed below the low speed area (hereinafter, the low speed area state). Here, the low speed area state includes a stop state in which the vehicle 1 stops and a low speed driving state in which the vehicle 1 drives in the low speed area. The low speed area of ​​the present embodiment refers to a range less than the allowable upper limit speed required to ensure the detection accuracy of the projection error PE. The value of the allowable upper limit speed can be changed in consideration of the image measurement specifications such as the frame rate and shutter speed in the ranging sensor 10 and the external camera 20, for example, values ​​such as 10km / h and 20km / h can be adopted. That is, in this modification example 1, the projection error detection unit 44 detects the projection error PE between the background light image ImL and the camera image ImC not only in the stop state, but also in the case of generating the background light image ImL and the camera image ImC in the low speed driving state.

[0097] As a second modification, the abnormality determination unit 45 may not determine the plurality of determination conditions, and may determine only one of the determination conditions shown in the first embodiment, for example.

[0098] As a third modification, the camera image ImC may be a grayscale image instead of a color image.

[0099] As a modification 4, at least one of the image processing ECU 30 and the driving assistance ECU may not be mounted on the vehicle, but may be fixedly installed on the road outside the vehicle, or may be mounted on another vehicle. In this case, abnormality detection, driving operation, etc. may also be remotely controlled through communications such as the Internet, road-to-vehicle communication, and vehicle-to-vehicle communication.

[0100] As a fifth modification, the object recognition abnormality detection device such as the image processing ECU 30 and the object recognition device such as the driving support ECU 50 may be integrated into one, for example, to constitute an electronic control device that realizes one complex function.

[0101] As a sixth modification, the abnormality detection device for object recognition may be a device that does not have an image registration function and is dedicated to abnormality detection.

[0102] As a modification 7, each function provided by the image processing ECU 30 may be provided by software and hardware executing the software, software alone, hardware alone, or a combination thereof. Furthermore, when such functions are provided by an electronic circuit as hardware, each function may be provided by a digital circuit including a plurality of logic circuits or an analog circuit.

[0103] As a modification 8, the form of the storage medium storing the abnormality detection program that can realize the above-mentioned abnormality detection method for object recognition can also be appropriately changed. For example, the storage medium is not limited to the structure provided on the circuit substrate, but can also be provided in the form of a memory card, etc., inserted into the socket, and electrically connected to the control circuit of the image processing ECU 30. In addition, the storage medium can also be a CD or a hard disk that becomes the copy basis of the program of the image processing ECU 30.

[0104] The control unit and method described in the present invention may also be implemented by a special-purpose computer, which is a processor programmed to execute one or more functions embodied by a computer program. Alternatively, the device and method described in the present invention may also be implemented by a special-purpose hardware logic circuit. Alternatively, the device and method described in the present invention may also be implemented by one or more special-purpose computers, which are composed of a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may also be stored as an instruction executed by a computer in a non-migratable tangible recording medium that can be read by a computer.

Claims

1. An abnormality detection device for object recognition, detecting abnormalities in the recognition of surrounding objects of a vehicle using a distance measuring sensor and an external camera, wherein the distance measuring sensor is mounted on the vehicle and generates a reflected light image containing distance information by sensing reflected light reflected from an object by light irradiation through a light receiving element, and generates a background light image having the same coordinate system as the reflected light image by sensing background light relative to the reflected light through the light receiving element, wherein the external camera is mounted on the vehicle and generates a camera image by detecting incident light from the outside through a camera element, wherein: The object recognition anomaly detection device comprises: a speed state grasping unit for grasping a speed state of the vehicle; An image information acquisition unit, which acquires the background light image and the camera image; as well as The projection error detection unit detects a projection error between the background light image and the camera image when the background light image and the camera image are generated in the speed state where the vehicle is less than a predetermined speed range.

2. The object recognition anomaly detection device according to claim 1, wherein: The projection error detection unit detects the projection error when the background light image and the camera image are generated in a stopped state where the vehicle is stopped.

3. The object recognition anomaly detection device according to claim 1, wherein: The projection error detection unit obtains corresponding points between the background light image and the camera image. The projection error detection unit determines the three-dimensional coordinates of the corresponding point on the background light image side by referring to the distance information. The projection error detection unit projects the three-dimensional coordinates onto a projection position on an image plane of the camera image. The projection error detection unit detects a deviation between a position of the corresponding point on the camera image side and the projection position as the projection error.

4. The object recognition anomaly detection device according to claim 1, wherein: The object recognition abnormality detection device further includes an abnormality determination unit configured to determine abnormality in the peripheral object recognition based on the projection error detected by the projection error detection unit.

5. The object recognition anomaly detection device according to claim 4, wherein: When the projection errors greater than a predetermined value that are greater than a predetermined number of detections are detected in the same group of the reflected light image and the camera image, the abnormality determination unit determines that there is an abnormality in the peripheral object recognition.

6. The object recognition anomaly detection device according to claim 4, wherein: The abnormality determination unit determines that there is an abnormality in the peripheral object recognition when an average value of the projection errors of a predetermined plurality of times detected most recently is equal to or larger than a predetermined value.

7. The object recognition anomaly detection device according to claim 4, wherein: When the projection error continues to increase, the abnormality determination unit determines that there is an abnormality in the peripheral object recognition.

8. The object recognition anomaly detection device according to any one of claims 4 to 7, wherein: The object recognition anomaly detection device further includes an anomaly notification information output unit that outputs anomaly notification information to an output destination outside the device when the anomaly determination unit determines that an anomaly exists.

9. A storage medium storing an object recognition anomaly detection program for detecting anomalies in recognition of surrounding objects of a vehicle, wherein: The storage medium includes instructions executed by a computer, and the instructions are used to cause at least one processing unit to perform the following processing: A process of acquiring a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor mounted on the vehicle, wherein the reflected light image containing distance information is generated by sensing reflected light reflected from an object by light irradiation through a light receiving element, and the background light image having the same coordinate system as the reflected light image is generated by sensing background light relative to the reflected light through the light receiving element; A process of acquiring a camera image, wherein the camera image is an image generated by an external camera mounted on the vehicle, the camera image being generated by detecting incident light from the outside world by a camera element; and A process of detecting a projection error between the background light image and the camera image when the background light image and the camera image are generated in a state where the speed of the vehicle is less than a predetermined speed range.

10. A method for detecting anomalies in object recognition, for detecting anomalies in the recognition of surrounding objects of a vehicle, wherein: The processing performed by at least one processing unit includes: Acquire a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor mounted on the vehicle, the reflected light image containing distance information is generated by a light receiving element sensing reflected light reflected from an object by light irradiation, and the background light image having the same coordinate system as the reflected light image is generated by the light receiving element sensing background light relative to the reflected light; Acquiring a camera image, wherein the camera image is an image generated by an external camera mounted on the vehicle, and the camera image is generated by detecting incident light from the outside world by a camera element; and When the background light image and the camera image are generated in a state where the speed of the vehicle is less than a predetermined speed range, a projection error between the background light image and the camera image is detected.

Images