Multi-sensor self-adaptive fault compensation system and method

Abstract

This technology overcomes the problem in conventional systems where the failure of a specific sensor within a redundant sensing system affects the object detection performance of the entire system. [Solution] The multi-sensor self-adaptive fault compensation system includes a plurality of sensors and a processor electrically connected to the plurality of sensors. The processor generates a first fusion result based on the sensor information of the plurality of sensors using a fusion module, determines the feature relationships between the sensor information of the plurality of sensors using a self-adaptive compensation module, determines whether the plurality of sensors are faulty or not using a fault determination module, generates sensor compensation information corresponding to the faulty sensor using a self-adaptive compensation module based on the sensor information of the active sensor and the feature relationships, and the fusion module generates a second fusion result based on the sensor information of the active sensor and the sensor compensation information.

Description

Technical Field

[0001] The present invention relates to a self - adaptive fault compensation system and method thereof, and particularly to a self - adaptive fault compensation system and method using multiple sensors.

Background Art

[0002] In the prior art, multiple sensors are mounted on an autonomous vehicle to sense the surrounding situation. Each of the multiple sensors has a different sensing function (for example, a camera, a radar, a lidar). Each of them independently performs object detection and outputs sensor information, and the sensor information includes information related to the object detection result (for example, the relative distance between the autonomous vehicle and the detected vehicle ahead). Also, since the sensing ranges of the multiple sensors overlap, a redundant sensing system can be configured. The multiple sensors transmit their respective sensor information to the processor of the autonomous vehicle, and the processor fuses the sensor information of the sensors with overlapping ranges using sensor fusion technology to generate a fusion result. Therefore, the information of the fusion result includes the relative distance between the autonomous vehicle and the vehicle ahead. As described above, since the multiple sensors are provided at different positions of the autonomous vehicle and identify objects based on different principles, the sensor information of the multiple sensors and the fusion result include the relative distance between the autonomous vehicle and the vehicle ahead, but they are not exactly equal.

[0003] Furthermore, in the prior art, the autonomous vehicle performs related control and judgment in autonomous driving based on the fusion result. With the redundant sensing system and sensor fusion technology, even if one sensor fails (for example, due to sensor freeze, weather influence, unstable signal transmission, or limitations of the sensing range), the operation of the sensor fusion can be ensured through the sensor information of other sensors.

[0004] However, with this method, if one sensor fails, even if the processor ensures the sensor fusion operation through sensor information from other sensors, the generated fusion result will already be affected. That is, for example, if the camera fails and the object detection function is handled only by the radar and optical radar, the camera will not output sensor information to the processor, so the fusion result generated by the processor using the sensor fusion technology will lack continuity, and furthermore, there is a risk that the processor may not be able to generate a fusion result at all due to the lack of sensor information from the camera. In that case, malfunctions may occur when the processor performs control and decision-making related to autonomous driving, and ultimately, there is a risk that the autonomous driving function may shut down. [Overview of the project] [Problems that the invention aims to solve]

[0005] The present invention aims to provide a multi-sensor self-adaptive fault compensation system and method in order to overcome the problem in the prior art where the failure of any sensor in a redundant sensing system affects the object detection performance of the entire system. [Means for solving the problem]

[0006] A multi-sensor self-adaptive fault compensation system according to one embodiment of the present invention is Multiple sensors, each outputting sensor information, The processor includes the plurality of sensors and is electrically connected to them. The aforementioned processor, A fusion module that generates the first or second fusion result, A fault determination module that connects the plurality of sensors and the fusion module, transmits sensor information from the plurality of sensors to the fusion module, and causes the fusion module to generate the first fusion result, The module includes a self-adaptive compensation module that connects the plurality of sensors, the fault detection module, and the fusion module, and determines the characteristic relationships between the sensor information of the plurality of sensors, When the fault detection module determines that one of the multiple sensors is faulty, it sends a fault message to the self-adaptive compensation module, defines the faulty sensor as a faulty sensor and the non-faulty sensors as valid sensors. When the self-adaptive compensation module receives the fault message, it generates sensor compensation information corresponding to the faulty sensor based on the sensor information of the active sensor and the characteristic relationship. The self-adaptive compensation module transmits the sensor information of the active sensor and the sensor compensation information to the fusion module, causing the fusion module to generate the second fusion result.

[0007] A multi-sensor self-adaptive fault compensation method according to one embodiment of the present invention is a method performed by a processor electrically connected to a plurality of sensors, To generate a first fusion result based on the sensor information of the aforementioned multiple sensors, Determining the characteristic relationships between the sensor information of the aforementioned multiple sensors, The system determines whether the aforementioned multiple sensors are malfunctioning, and generates sensor compensation information corresponding to the malfunctioning sensor based on the sensor information of the effective sensor and the characteristic relationship. This includes generating a second fusion result based on the sensor information of the effective sensor and the sensor compensation information.

[0008] According to this configuration and method, the present invention determines the characteristic relationship between the sensor information of a sensor that has not yet failed and the first fusion result at that time. If a sensor fails, it generates sensor compensation information based on the sensor information of an active sensor and the aforementioned characteristic relationship, thereby achieving the effect of self-adaptive compensation. Therefore, the present invention can improve object detection function and stability, and reduce discontinuities in output information, thereby achieving driving stability. Furthermore, it can prevent the system from shutting down its automatic driving function and ensure driving safety, stability, and comfort. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic block diagram of an embodiment of the multi-sensor self-adaptive fault compensation system of the present invention. [Figure 2] This is a schematic top view diagram of a plurality of sensors provided on a vehicle and their sensing ranges according to the present invention. [Figure 3A] This is a schematic diagram (1) of the waveform obtained from the simulation of the present invention. [Figure 3B] This is a schematic diagram (2) of the waveform of the simulation results of the present invention. [Figure 3C] This is a schematic diagram (3) of the waveform of the simulation result of the present invention. [Figure 4] This is a flowchart of the first embodiment showing the flow of fault message generation by the processor in the present invention. [Figure 5] This is a flowchart for performing tracking and matching of a target object in the present invention. [Figure 6] This is a flowchart of a second embodiment showing the flow of fault message generation by the processor in the present invention. [Figure 7] This is a flowchart of a third embodiment illustrating the process of generating fault messages by the processor in the present invention. [Figure 8] This is a flowchart (1) showing the flow in which the processor performs self-adaptive compensation in the present invention. [Figure 9]This is a flowchart (2) showing the flow in which the processor performs self-adaptive compensation in the present invention. [Modes for carrying out the invention]

[0010] As shown in Figures 1 and 2, the multi-sensor self-adaptive fault compensation system of the present invention is applied to a vehicle V and includes a plurality of sensors 10, storage 20, and a processor 30, and the vehicle V may be an autonomous vehicle. The plurality of sensors 10 are each installed at several locations on the vehicle V so as to sense the surrounding conditions of the vehicle V. The storage 20 is installed on the vehicle V and can be, for example, a hard disk drive (HDD), solid state drive (SSD), memory, memory card, etc. The processor 30 can be installed in the vehicle control unit (VCU) or electronic control unit (ECU) of the vehicle V and is electrically connected to the plurality of sensors 10 and storage 20. In addition, each sensor 10 independently performs object detection and outputs sensor information 100 to the processor 30. The sensor information 100 includes the object detection result, for example, the object detection result of the sensor information 100 includes the vehicle in front of the vehicle V (hereinafter referred to as the target object FV) and the relative distance between the target object FV and the vehicle V.

[0011] Furthermore, for example, to sense the situation in front of vehicle V, two or more of the following sensors 10 are used: a first corner radar sensor 11, a second corner radar sensor 12, a forward radar sensor 13, an optical radar sensor 14 (Light Detection and Ranging, LIDAR), and a forward-view camera 15. The first corner radar sensor 11 and the second corner radar sensor 12 are located on the left and right sides of the front of vehicle V, respectively.

[0012] Since the plurality of sensors 10 each have their own field of view, when attaching the plurality of sensors 10 to the vehicle V, each sensor 10 can be attached to an appropriate position according to demand. In that case, among the plurality of sensors 10, the fields of view of any two of them may overlap. In the storage 20, a matching table is stored, and default data for defining the matching states of the plurality of sensors 10 is preset in the matching table. For any two sensors 10 that are matched, their respective fields of view overlap. Taking FIG. 2 as an example, since the field of view 110 of the first corner radar sensor 11, the field of view 130 of the front radar sensor 13, the field of view 140 of the optical radar sensor 14, and the field of view 150 of the front view camera 15 overlap, they are matched with each other. Similarly, since the field of view 120 of the second corner radar sensor 12, the field of view 130 of the front radar sensor 13, the field of view 140 of the optical radar sensor 14, and the field of view 150 of the front view camera 15 also overlap, they are matched with each other. However, since the fields of view 110 and 120 of the first corner radar sensor 11 and the second corner radar sensor 12 do not overlap, they are not matched with each other. The processor 30 reads the matching table from the storage 20 and determines the matching states between the plurality of sensors 10 by referring to the table.

[0013] Furthermore, the processor 30 can execute the programs of the fusion module 31, the fault determination module 32, and the self - adaptive compensation module 33. The programs of the fusion module 31, the fault determination module 32, and the self - adaptive compensation module 33 are stored in the storage 20 so that the processor 30 can read and execute them. As shown in FIG. 1, the fault determination module 32 is connected to the plurality of sensors 10 and the fusion module 31, and the self - adaptive compensation module 33 is connected to the plurality of sensors 10, the fault determination module 32, and the fusion module 31.

[0014] The processor 30 generates either a first fusion result Z1 or a second fusion result Z2 using the fusion module 31. The first fusion result Z1 is generated by the fusion module 31 based on sensor information 100 from multiple sensors 10 (i.e., when all multiple sensors 10 are active), and the second fusion result Z2 is generated by the fusion module 31 based on sensor information 100 from partially active sensors 10 (i.e., when at least one sensor is faulty) and sensor compensation information C. The information in the first fusion result Z1 and the information in the second fusion result Z2 each include information regarding the object detection result (i.e., the target object FV and the relative distance between the vehicle V and the target object FV). Such sensor fusion technology is common knowledge in its field and will not be explained in detail.

[0015] Furthermore, the processor 30 can determine whether or not any one of the multiple sensors 10 is faulty using the fault detection module 32. In this invention, three embodiments are given for the processor 30 to determine whether or not the multiple sensors 10 are faulty, and these embodiments will be described later. If the processor 30 determines that one of the multiple sensors 10 is faulty using the fault detection module 32, the fault detection module 32 sends a fault message S1 to the self-adaptive compensation module 33. This fault message S1 may also be a fault flag. On the other hand, if the fault detection module 32 determines that all of the multiple sensors 10 are valid, the processor 30 sends the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault detection module 32, causing the fusion module 31 to generate a first fusion result Z1, and saves the first fusion result Z1 to the storage 20. Furthermore, the processor 30 determines the characteristic relationships of the sensor information 100 of each sensor 10 using the self-adaptive compensation module 33. The feature relationship includes (1) information on the difference in object detection results between the sensor information 100 of each sensor 10 and the sensor information 100 of other sensors 10, and (2) information on the difference in object detection results between the sensor information 100 of each sensor 10 and the first fusion result Z1. Furthermore, the feature relationship may also include information on the difference in object detection results between the sensor information 100 of each sensor 10 and the second fusion result Z2.

[0016] Hereinafter, for the convenience of explanation, among the plurality of sensors 10, those that are malfunctioning are defined as faulty sensors, and those that are not malfunctioning are defined as valid sensors. When the self-adaptive compensation module 33 receives a fault message S1, the processor 30 generates sensor compensation information C based on the sensor information 100 of the valid sensors and the characteristic relationship. The sensor compensation information C includes the target object FV and the relative distance between the vehicle V and the target object FV. In this way, the sensor compensation information C can compensate for the missing sensor information of the faulty sensor according to the faulty sensor. Then, when the self-adaptive compensation module 33 transmits the sensor information 100 of the valid sensors and the sensor compensation information C to the fusion module 31, the processor 30 causes the fusion module 31 to generate a second fusion result Z2 and stores the second fusion result Z2 in the storage 20.

[0017] In this way, the processor 30 can perform related control and judgment for autonomous driving based on the first fusion result Z1 or the second fusion result Z2. That is, to stabilize the autonomous driving state and prevent the autonomous driving function from being affected or interrupted unexpectedly, the first fusion result Z1 is adopted when it is determined that all the sensors 10 are valid, and the second fusion result Z2 is adopted when it is determined that at least one sensor 10 is malfunctioning.

[0018] The following describes an example of the simulation results of the present invention. In an embodiment in which multiple sensors 10 include a foresight camera 15 and a forward radar sensor 13, the first waveform W1 shown in Figure 3A is the waveform of the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the foresight camera 15, the second waveform W2 shown in Figure 3A is the waveform of the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward radar sensor 13, and the third waveform W3 shown in Figure 3A is the waveform of the relative distance between the vehicle V and the target object FV included in the information of the first fusion result Z1. Note that the foresight camera 15 and the forward radar sensor 13 are installed at different locations on the vehicle V and the principles of object identification are also different, so the relative distances shown by the first waveform W1, the second waveform W2, and the third waveform W3 do not perfectly match, but the trends match.

[0019] As described above, the processor 30 determines the feature relationships of the sensor information 100 of each sensor 10 using the self-adaptive compensation module 33. As shown in Figure 3A, at any given time tx, the feature relationships corresponding to the foresight camera 15 include a first relative distance difference Da and a second relative distance difference Db. The first relative distance difference Da is the difference information (i.e., the difference value between the first waveform W1 and the second waveform W2 at time tx) between the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the foresight camera 15 (i.e., the first waveform W1) and the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward radar sensor 13 (i.e., the second waveform W2). The second relative distance difference Db is the difference information (i.e., the difference value between the first waveform W1 and the second waveform W2 at time tx). 1) The difference information between the first fusion result Z1 and the relative distance between the vehicle V and the target object FV (i.e., the third waveform W3) included in the first fusion result Z1 (i.e., the difference value between the first waveform W1 and the third waveform W3 at time tx). The feature relationship corresponding to the forward radar sensor 13 includes the difference Da of the first relative distance and the difference Dc of the third relative distance. The difference Dc of the third relative distance refers to the difference information between the relative distance between the vehicle V and the target object FV (i.e., the second waveform W2) included in the sensor information 100 of the forward radar sensor 13 and the relative distance between the vehicle V and the target object FV (i.e., the third waveform W3) included in the first fusion result Z1 (i.e., the difference value between the third waveform W3 and the second waveform W2 at time tx). Similarly, the processor 30 can also determine, by the self-adaptive compensation module 33, that the feature relationship corresponding to the first fusion result Z1 includes the difference Db of the second relative distance and the difference Dc of the third relative distance.

[0020] As shown in Figure 3B, the first waveform W1 has an interruption interval W1_loss between 12 and 18 seconds, simulating that the foresight camera 15 failed during the interruption interval W1_loss. The third waveform W3 can be seen to approach and overlap with the second waveform W2 during the interruption interval W1_loss. This result indicates that the first fusion result Z1 is almost entirely dependent on the sensor information 100 from the forward radar sensor 13.

[0021] As shown in Figure 3C, according to the self-adaptive compensation means of the present invention, in the interruption interval W1_loss of the first waveform W1 shown in Figure 3B, the waveform of the sensor compensation information C is supplemented, and the continuity of the first waveform W1 can be maintained. Furthermore, the entirety of the third waveform W3 shown in Figure 3C approaches the third waveform W3 in Figure 3A, reflecting the compensation results of the sensor compensation information C.

[0022] The process of the multi-sensor self-adaptive fault compensation method of the present invention will be described in detail below.

[0023] As shown in Figure 4, when the processor 30 receives sensor information 100 from multiple sensors 10, it checks whether a first fusion result Z1 exists (step S01). For example, each time the fusion module 31 generates a first fusion result Z1, it generates a flag corresponding to this result, and the processor 30 checks whether a first fusion result Z1 exists based on this flag.

[0024] If the result of step S01 is "NO", the processor 30 proceeds to determine whether or not the multiple sensors 10 have frozen (step S02). For example, each sensor information 100 transmitted from each sensor 10 to the processor 30 may include a count value, and normally, the difference between the count values ​​of two pieces of sensor information 100 received by the processor 30 from each sensor 10 before and after is not zero. Therefore, if the difference between the count values ​​of two pieces of sensor information 100 received by the processor 30 from a certain sensor 10 before and after is zero, the processor 30 can determine that the sensor 10 has frozen. On the other hand, if step S02 is "NO", it indicates that none of the multiple sensors 10 have frozen, in which case the processor 30 transmits the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault determination module 32 (step S03), thereby causing the fusion module 31 to generate the first fusion result Z1.

[0025] The following describes a first embodiment in which the processor 30 generates a fault message S1 using the fault determination module 32. In this embodiment, for ease of understanding, the multiple sensors 10 include a first sensor and a second sensor, and the first sensor and the second sensor are matched with each other in the matching table of the storage 20. As shown in Figure 4, if the determination result of step S01 is "YES", the processor 30 uses the fault determination module 32 to perform target object tracking matching on the sensor information 100 of the multiple sensors 10 against the first fusion result Z1 (step S04), thereby determining whether the sensor information 100 of the multiple sensors 10 and the first fusion result Z1 recognize the same target object FV. Furthermore, the target object tracking matching process shown in Figure 5 is common knowledge in the art to which it belongs, and simply put, the processor 30 executes a program of a Kalman filter and generates a first predicted fusion result based on the first fusion result Z1 (step S041). Next, the data association algorithm is executed, for example, by running a Hungarian algorithm program to generate a matching result based on the first predicted fusion result and the sensor information of the first sensor (step S042). If this matching result is, for example, True or 1, it indicates that the matching result for the first sensor was successful, and in this case, the processor 30 can recognize the same target object FV from the sensor information of the first sensor and the first predicted fusion result. Conversely, if the matching result is, for example, False or 0, it indicates that the matching for the first sensor failed, and the processor 30 cannot recognize the same target object FV from the sensor information of the first sensor and the first predicted fusion result. The second sensor is processed similarly, and the Kalman filter calculation program and the Hungarian algorithm program described above are stored in the storage 20 so that the processor 30 can access and execute them.

[0026] If both the first and second sensors successfully match, the processor 30 transmits sensor information 100 from the multiple sensors 10 to the fusion module 31 via the fault detection module 32, causing the fusion module 31 to generate the first fusion result for the next time point, and also saves that first fusion result for the next time point to the storage 20. On the other hand, if the first sensor fails to match, the processor 30 uses the fault detection module 32 to determine whether the target object FV in the first predicted fusion result is located within the sensing range of the first sensor (S05). Furthermore, since the processor 30 generates spatial coordinate information around the vehicle V, the position of the vehicle V, the position of the target object FV, and the sensing ranges of the multiple sensors 10 can be identified by this coordinate information. Note that these are common knowledge in the relevant technical field and will not be explained in detail. In this way, the processor 30 determines, based on the coordinate information, whether the target object FV in the first predicted fusion result is located within the sensing range of the first sensor, and the fault detection module 32 determines whether the sensing range of the first sensor overlaps with the sensing range of the second sensor (step S05). Furthermore, the processor 30 can determine whether the sensing range of the first sensor overlaps with the sensing range of the second sensor based on the matching table stored in the storage 20.

[0027] Furthermore, if the judgment result in step S05 is "NO", it indicates that the target object FV in the first predicted fusion result is not within the detection range of the first sensor, and / or the detection range of the first sensor and the detection range of the second sensor do not overlap. However, this may be because the target object FV has moved outside the detection range of some of the sensors 10. Therefore, the processor 30 continues to transmit sensor information 100 from multiple sensors 10 to the fusion module 31 via the fault judgment module 32 (step S03), causing the fusion module 31 to generate the first fusion result for the next point in time.

[0028] On the other hand, if the result of the judgment in step S05 is "YES", it indicates that the target object FV in the first predicted fusion result is within the detection range of the first sensor and the second sensor, and that the detection range of the first sensor and the detection range of the second sensor overlap. In other words, if the sensor information of the second sensor includes the target object FV, but the sensor information of the first sensor does not include the target object FV, the processor 30 determines the first sensor to be a faulty sensor and the second sensor to be a valid sensor, and the fault judgment module 32 sends a fault message S1 to the self-adaptive compensation module 33 (step S06).

[0029] The following describes a second embodiment in which the processor 30 generates a fault message S1 using the fault detection module 32. As shown in Figure 6, the sensor information 100 that each sensor 10 outputs to the processor 30 may include credibility information, which is, for example, True or False, indicating whether or not the object detection result is credible. In this way, the processor 30 uses the fault detection module 32 to determine whether or not the credibility information of each of the multiple sensors 10 is credible (step S07). If the result of the determination in step S07 is "NO", it indicates that there is some sensor information 100 of a sensor 10 that lacks credibility. In that case, as described in step S05 above, the processor 30 uses the fault detection module 32 to determine each unreliable sensor 10 as a faulty sensor and each credible sensor 10 as a valid sensor, and the fault detection module 32 sends a fault message S1 to the self-adaptive compensation module 33. On the other hand, if the result of the judgment in step S07 is "YES", it means that the sensor information 100 from all of the sensors 10 is reliable, so the processor 30 transmits the sensor information 100 from all of the sensors 10 to the fusion module 31 via the fault judgment module 32 (step S03), causing the fusion module 31 to generate the first fusion result for the next time point.

[0030] The following describes a third embodiment in which the processor 30 generates the fault message S1 using the fault determination module 32. As shown in Figure 7, the third embodiment is a combination of the first and second embodiments. If the determination result of step S01 is "YES", the processor 30 further uses the fault determination module 32 to determine whether the credibility information of each of the multiple sensors 10 is credible (step S07'). On the other hand, if the determination result of step S07' is "NO", the processor 30 transmits the fault message S1 to the self-adaptive compensation module 33 via the fault determination module 32 (step S06'). Subsequently, if the determination result of step S07' is "YES", i.e., credible, the processor 30 uses the fault determination module 32 to perform tracking matching of the sensor information 100 of each of the multiple sensors 10 against the first fusion result Z1 (step S04'). As in the previous embodiment, the multiple sensors include a first sensor and a second sensor. If both the first sensor and the second sensor successfully match, the processor 30 transmits the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault detection module 32 (step S03), causing the fusion module 31 to generate the first fusion result for the next time point. On the other hand, if the matching of the first sensor fails, the processor 30 has the fault detection module 32 execute step S05 as described above.

[0031] The above describes three embodiments in which the processor 30 generates the fault message S1 using the fault detection module 32. Next, we will describe the processing performed by the processor 30 and the self-adaptive compensation module 33 when the fault message S1 has not been received, and after the fault message S1 has been received.

[0032] As shown in Figure 8, the processor 30, using the self-adaptive compensation module 33, determines whether or not it has received a fault message S1 from the fault detection module 32 (step S11). If the result of the determination in step S11 is "NO", the processor 30 indicates that the fault detection module 32 did not detect the fault sensor 10. The processor 30 then determines whether or not a first fusion result Z1 exists (step S12, see step S01 described above). Subsequently, if the result of the determination in step S12 is "YES", the processor 30, using the self-adaptive compensation module 33, extracts the above feature relationship based on the first fusion result Z1 and the sensor information 100 of the multiple sensors 10 (step S13) and updates the feature relationship information (step S14). On the other hand, if the result of the determination in step S12 is "NO", the processor 30, using the self-adaptive compensation module 33, reads the default feature relationship information from the storage 20 (step S15) and updates it. The purpose of these steps S13 to S15 is to track the target object FV, which is traveling in front of the vehicle V, in real time using the updated feature relationship information.

[0033] As shown in Figure 9, if the result of step S11 is "YES", the processor 30 uses the self-adaptive compensation module 33 to determine whether all of the matching sensors 10 are faulty (step S16), and as described above, the processor 30 determines whether both the first sensor and the second sensor are faulty sensors. If the result of step S16 is "YES", it indicates that the target object FV may have moved out of the sensing range of both the first sensor and the second sensor, and does not indicate that both the first sensor and the second sensor have failed simultaneously. In this case, the processor 30 transmits the current sensor information 100 of the multiple sensors 10 to the fusion module 31 via the self-adaptive compensation module 33 (step S17), causing the fusion module 31 to generate the first fusion result for the next time point.

[0034] On the other hand, if the result of the judgment in step S16 is "NO", it indicates that only some of the sensors 10 are able to detect the target object FV, that is, as described above, only the second sensor (effective sensor) is detecting the target object FV, and the first sensor (faulty sensor) is not detecting the target object FV. In this case, the processor 30 further determines whether or not a first fusion result from the previous point in time exists (step S18). Here, the processor 30 may save the previously generated first fusion result to the storage 20 via the fusion module 31 and define it as the first fusion result from the previous point in time.

[0035] If the result of the judgment in step S18 is "YES", the processor 30 uses the self-adaptive compensation module 33 to make a prediction based on the first fusion result from the previous time point and generates information on the predicted fusion result (step S19). The processor 30 also uses the self-adaptive compensation module 33 to determine that the feature relationship corresponding to the predicted fusion result includes the target object FV and the predicted relative distance between the target object FV and the vehicle V. At this time, the processor 30 performs a Kalman filter operation to make a prediction based on the first fusion result from the previous time point and generates the predicted fusion result. Since the aforementioned feature relationship includes the feature relationship between the predicted fusion result and the sensor information 100 when the multiple sensors 10 are not malfunctioning, the processor 30 uses the self-adaptive compensation module 33 to generate sensor compensation information C based on the sensor information 100 of the active sensors and the feature relationship (step S20).

[0036] Following the previous embodiment, if neither the foresight camera 15 nor the forward radar sensor 13 is malfunctioning, the first relative distance difference Da is 1.3 meters, the second relative distance difference Db is 1.1 meters, and the third relative distance difference Dc is 0.2 meters. If, at the next point in time, the foresight camera 15 malfunctions and becomes a faulty sensor, but the forward radar sensor 13 is still active, the processor 30 generates information for the prediction fusion result. If this prediction fusion result corresponds to the prediction waveform W3_predict shown in Figure 3C, that is, the third relative distance difference when the foresight camera 15 malfunctions, is the prediction difference information (e.g., 0.3 meters) between the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward radar sensor 13 (active sensor) (i.e., the second waveform W2) and the prediction relative distance between the vehicle V and the target object FV included in the prediction fusion result (i.e., the prediction waveform W3_predict). When the difference value of this predicted information (0.3 meters) is added to the second relative distance difference Db (1.1 meters) when both the foresight camera 15 and the forward radar sensor 13 are functioning correctly, the first relative distance difference when the foresight camera 15 fails is predicted to be 1.4 meters (i.e., 0.3 + 1.1 = 1.4), and sensor compensation information C is generated based on this predicted first relative distance.

[0037] If the result of the judgment in step S18 is "NO", the processor 30 reads the default fusion result information from the storage 20 using the self-adaptive compensation module 33. At this time, the feature relationship described above is the feature relationship between the default fusion result and the sensor information 100 when the multiple sensors 10 are not malfunctioning, and the processor 30 generates sensor compensation information C using the self-adaptive compensation module 33 based on the sensor information 100 of the active sensors and this feature relationship.

[0038] After the sensor compensation information C is generated, the processor 30 further determines whether there is any missing sensor information using the self-adaptive compensation module 33 (step S21). For example, as shown in Figure 3B, if the processor 30 determines that the first waveform W1 is a discontinuous waveform because there is an interruption interval W1_loss in the first waveform W1, it determines that there is missing sensor information. If the result of the determination in step S21 is "NO", it means that the missing sensor information has been compensated by the sensor compensation information C. Also, in Figure 3C, the sensor compensation information C is already included in the first waveform W1, that is, the interruption interval W1_loss has been replaced by the sensor compensation information C. In this case, the processor 30 continues to transmit the current sensor information 100 of the multiple sensors 10 to the fusion module 31 via the self-adaptive compensation module 33 (step S22), and the self-adaptive compensation module 33 performs feature relationship extraction based on the second fusion result Z2 (step S23), and updates the feature relationship information with the feature relationships extracted from the second fusion result Z2 (step S14).

[0039] As described above, the present invention has the following effects.

[0040] 1. In response to the tracking matching of the target object in step S04 above, the processor 30 tracks the target object FV based on the prediction of the first fusion result Z1, and determines whether each sensor 10 has lost track of the target object FV, and further determines whether a sensor has malfunctioned.

[0041] Second, the processor 30 determines the feature relationship (distance) that exists between the first fusion result at the previous point in time, the current first fusion result Z1, and the current sensor information 100 of each sensor 10. If a sensor fails, it uses this feature relationship to automatically generate the missing sensor data, i.e., sensor compensation information C, in a self-adaptive manner and provides it to the fusion module 31, thereby avoiding problems such as deviations in the relative distance of target detection (discontinuity in output) and inability to detect.

[0042] 3. The processor 30 improves the stability of object detection and tracking through the collaborative operation of the fusion module 31, the fault judgment module 32, and the self-adaptive compensation module 33. [Explanation of symbols]

[0043] 10 sensors 11. First corner radar sensor 12. Second corner radar sensor 13. Forward radar sensor 14 Optical radar sensor 15 Foreview Camera 20 storage 30 processors 31 Fusion Modules 32. Fault detection module 33 Self-adaptive compensation module 100 Sensor Information 110, 120, 130, 140, 150 detection range V Vehicle FV Target Object Z1 first fusion result Z2 second fusion result C Sensor Compensation Information S1 Error Message W1 1st waveform W1_loss Interruption interval W2 Second waveform W3 third waveform W3_predict Predicted waveform Da First Relative Distance Difference Db Second relative distance difference Dc Third relative distance difference tx time

Claims

1. A multi-sensor self-adaptive fault compensation system, Multiple sensors, each outputting sensor information, The processor includes the plurality of sensors and is electrically connected to them. The aforementioned processor, A fusion module that generates the first or second fusion result, A fault determination module that connects the plurality of sensors and the fusion module, transmits sensor information from the plurality of sensors to the fusion module, and causes the fusion module to generate the first fusion result, The module includes a self-adaptive compensation module that connects the plurality of sensors, the fault detection module, and the fusion module, and determines the characteristic relationships between the sensor information of the plurality of sensors, When the fault detection module determines that one of the multiple sensors is faulty, it sends a fault message to the self-adaptive compensation module, defines the faulty sensor as a faulty sensor and the non-faulty sensors as valid sensors. When the self-adaptive compensation module receives the fault message, it generates sensor compensation information corresponding to the faulty sensor based on the sensor information of the active sensor and the characteristic relationship. A multi-sensor self-adaptive fault compensation system characterized in that the self-adaptive compensation module transmits sensor information of the active sensor and sensor compensation information to the fusion module, causing the fusion module to generate the second fusion result.

2. The plurality of sensors include a first sensor and a second sensor, The fault detection module performs tracking matching of the target object with respect to the first fusion result using the sensor information of each of the multiple sensors. If the first sensor and the second sensor successfully match, the fault detection module transmits sensor information from the multiple sensors to the fusion module, causing the fusion module to generate a first fusion result for the next time point. On the other hand, if the first sensor fails to match, the fault detection module determines whether the target object in the first predicted fusion result is located within the detection range of the first sensor, and whether the detection range of the first sensor overlaps with the detection range of the second sensor. If "NO" is determined, the fault detection module transmits the sensor information of the multiple sensors to the fusion module, causing the fusion module to generate the first fusion result for the next time point. On the other hand, if "YES" is determined, the fault determination module determines the first sensor to be the fault sensor and transmits the fault message to the self-adaptive compensation module, characterized in that the multi-sensor self-adaptive fault compensation system according to claim 1.

3. The sensor information output by each of the aforementioned sensors includes credibility information. The fault detection module determines whether the reliability information of each of the multiple sensors is reliable or not. If "NO" is determined, the fault detection module determines that the sensor is unreliable and transmits the fault message to the self-adaptive compensation module. On the other hand, if "YES" is determined, the multi-sensor self-adaptive fault compensation system according to claim 1, characterized in that the fault determination module transmits sensor information from the plurality of sensors to the fusion module, causing the fusion module to generate a first fusion result for the next time point.

4. The sensor information output by each of the aforementioned sensors includes credibility information. The fault detection module determines whether the reliability information of each of the multiple sensors is reliable or not. If "NO" is determined, the fault detection module determines that the sensor is unreliable and transmits the fault message to the self-adaptive compensation module. On the other hand, if "YES" is determined, the fault determination module performs tracking matching of the target object with respect to the first fusion result using the sensor information of each of the multiple sensors, and the multiple sensors include the first sensor and the second sensor, If the first sensor and the second sensor successfully match, the fault detection module transmits sensor information from the multiple sensors to the fusion module, causing the fusion module to generate a first fusion result for the next time point. On the other hand, if the first sensor fails to match, the fault detection module determines whether the target object in the first predicted fusion result is located within the detection range of the first sensor, and whether the detection range of the first sensor overlaps with the detection range of the second sensor. If "NO" is determined, the fault detection module transmits the sensor information of the multiple sensors to the fusion module, causing the fusion module to generate the first fusion result for the next time point. On the other hand, if "YES" is determined, the fault determination module determines the first sensor to be the fault sensor and transmits the fault message to the self-adaptive compensation module, characterized in that the multi-sensor self-adaptive fault compensation system according to claim 1.

5. When the self-adaptive compensation module receives the fault message, it determines whether or not a first fusion result from an earlier point in time exists. If "YES" is determined, the self-adaptive compensation module makes a prediction based on the first fusion result from the previous time point and generates information of the predicted fusion result, and the feature relationship is the feature relationship between the information of the predicted fusion result and the sensor information of the multiple sensors. On the other hand, if "NO" is determined, the self-adaptive compensation module reads information of the default fusion result from storage, and the feature relationship is a feature relationship between the default fusion result and the sensor information of the plurality of sensors, as described in claim 1, for the multi-sensor self-adaptive fault compensation system.

6. A method performed by a processor electrically connected to multiple sensors, To generate a first fusion result based on the sensor information of the aforementioned multiple sensors, Determining the characteristic relationships between the sensor information of the aforementioned multiple sensors, The system determines whether the aforementioned multiple sensors are malfunctioning, and generates sensor compensation information corresponding to the malfunctioning sensor based on the sensor information of the effective sensor and the characteristic relationship. A multi-sensor self-adaptive fault compensation method characterized by generating a second fusion result based on the sensor information of the effective sensor and the sensor compensation information.

7. The aforementioned plurality of sensors include a first sensor and a second sensor. In the step of determining whether the aforementioned multiple sensors are malfunctioning, The sensor information from each of the aforementioned multiple sensors is used to perform tracking matching of the target object with respect to the first fusion result. If the first sensor and the second sensor successfully match, a first fusion result for the next time point is generated based on the sensor information of the multiple sensors. If the first sensor fails to match, it is determined whether the target object in the first predicted fusion result is located within the detection range of the first sensor, and whether the detection range of the first sensor overlaps with the detection range of the second sensor. If "NO" is determined, the first fusion result for the next time point is generated based on the sensor information of the multiple sensors. On the other hand, the multi-sensor self-adaptive fault compensation method according to claim 6, characterized in that if "YES" is determined, the first sensor is determined to be the faulty sensor.

8. The sensor information output by each of the aforementioned sensors includes credibility information. In the step of determining whether the aforementioned multiple sensors are malfunctioning, The reliability information of each of the aforementioned sensors is determined to be reliable or not. If "NO" is determined, the unreliable sensor is determined to be the faulty sensor. On the other hand, if it is determined to be "YES", the multi-sensor self-adaptive fault compensation method according to claim 6 is characterized by including generating a first fusion result for the next time point based on the sensor information of the plurality of sensors.

9. The sensor information output by each of the aforementioned sensors includes credibility information. In the step of determining whether the aforementioned multiple sensors are malfunctioning, The reliability information of each of the aforementioned sensors is determined to be reliable or not. If "NO" is determined, the unreliable sensor is determined to be the faulty sensor. On the other hand, if "YES" is determined, the sensor information of the multiple sensors is used to perform tracking matching of the target object with respect to the first fusion result, and the multiple sensors include the first sensor and the second sensor. If the first sensor and the second sensor successfully match, a first fusion result for the next time point is generated based on the sensor information of the multiple sensors. On the other hand, if the first sensor fails to match, it is determined whether the target object in the first predicted fusion result is located within the detection range of the first sensor, and whether the detection range of the first sensor overlaps with the detection range of the second sensor. If "NO" is determined, the first fusion result for the next time point is generated based on the sensor information of the multiple sensors. On the other hand, the multi-sensor self-adaptive fault compensation method according to claim 6, characterized in that if "YES" is determined, the first sensor is determined to be the faulty sensor.

10. If it is determined that one of the aforementioned sensors has malfunctioned, it is then determined whether or not a first fusion result from an earlier point in time exists. If "YES" is determined, a prediction is made based on the first fusion result from the previous point in time to generate information for the predicted fusion result, and the feature relationship is the feature relationship between the predicted fusion result and the sensor information of the multiple sensors. On the other hand, if "NO" is determined, the information of the default fusion result is read, and the feature relationship is a feature relationship between the default fusion result and the sensor information of the plurality of sensors, as described in claim 6.

Images