Multi-sensor self-adaptive fault compensation system and method

Abstract

This overcomes the problem in the prior art that failure of a specific sensor in a redundant sensing system affects the object detection performance of the entire system. [Solution] A multi-sensor self-adaptive fault compensation system includes a plurality of sensors and a processor electrically connected to the plurality of sensors, wherein the processor generates a first fusion result based on sensor information from the plurality of sensors using a fusion module, determines the characteristic relationship between the sensor information from the plurality of sensors using a self-adaptive compensation module, determines whether the plurality of sensors are faulty using a fault judgment module, generates sensor compensation information corresponding to the faulty sensor using the self-adaptive compensation module based on the sensor information of the valid sensors and the characteristic relationship, and the fusion module generates a second fusion result based on the sensor information from the valid sensors and the sensor compensation information.

Description

[Technical Field]

[0001] The present invention relates to a self-adaptive fault compensation system and method, and more particularly to a self-adaptive fault compensation system and method using multiple sensors. [Background technology]

[0002] In conventional technology, autonomous vehicles are equipped with multiple sensors to sense their surroundings, each with a different sensing function (e.g., camera, radar, optical radar). Each sensor independently detects objects and outputs sensor information, which includes information about the object detection result (e.g., the relative distance between the autonomous vehicle and the detected vehicle ahead). Furthermore, because the multiple sensors have overlapping sensing ranges, a redundant sensing system can be configured. The multiple sensors transmit their respective sensor information to a processor in the autonomous vehicle, which then fuses the sensor information from the overlapping sensors with object information using sensor fusion technology to generate a fusion result. Therefore, the fusion result information includes the relative distance between the autonomous vehicle and the vehicle ahead. As described above, because the multiple sensors are installed in different positions on the autonomous vehicle and use different principles to identify objects, the sensor information from the multiple sensors and the fusion result include the relative distance between the autonomous vehicle and the vehicle ahead, but are not necessarily identical.

[0003] Furthermore, in conventional technology, the autonomous vehicle performs relevant control and decision-making during autonomous driving based on the fusion results, and redundant sensing systems and sensor fusion technology ensure the operation of the sensor fusion through sensor information from other sensors even if one sensor fails (e.g., due to sensor freezing, weather effects, unstable signal transmission, or sensing range limitations).

[0004] However, with this method, if one sensor fails, even if the processor ensures the sensor fusion operation through sensor information from other sensors, the generated fusion results will already be affected. That is, for example, if the camera fails and the object detection function is performed solely by the radar and the optical radar, the camera will not output sensor information to the processor, and the fusion results generated by the processor using the sensor fusion technology will have no continuity. Furthermore, the processor may not be able to generate fusion results due to the lack of sensor information from the camera. This may cause malfunctions when the processor performs autonomous driving-related control and decision-making, ultimately resulting in the suspension of the autonomous driving function. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to provide a multi-sensor self-adaptive fault compensation system and method therefor to overcome the problem in the prior art that the failure of any sensor in a redundant sensing system will affect the object detection performance of the entire system. [Means for solving the problem]

[0006] A multi-sensor self-adaptive fault compensation system according to an embodiment of the present invention comprises: A plurality of sensors each outputting sensor information; a processor electrically connected to the plurality of sensors; The processor: a fusion module that generates a first fusion result or a second fusion result; a fault judgment module connecting the plurality of sensors and the fusion module, transmitting sensor information of the plurality of sensors to the fusion module, and causing the fusion module to generate the first fusion result; a self-adaptive compensation module connected to the plurality of sensors, the fault determination module, and the fusion module, for determining a feature relationship between sensor information of the plurality of sensors; When the fault determination module determines that one of the plurality of sensors is faulty, it sends a fault message to the self-adaptive compensation module, and defines the faulty sensors of the plurality of sensors as faulty sensors and the non-faulty sensors as valid sensors; When the self-adaptive compensation module receives the fault message, it generates sensor compensation information corresponding to the faulty sensor according to the sensor information of the valid sensor and the characteristic relationship; The self-adaptive compensation module sends the sensor information of the available sensors and the sensor compensation information to the fusion module, causing the fusion module to generate the second fusion result.

[0007] A multi-sensor self-adaptive fault compensation method according to an embodiment of the present invention is a method performed by a processor electrically connected to a plurality of sensors, the method comprising: generating a first fusion result based on sensor information of the plurality of sensors; determining a feature relationship between the sensor information of the plurality of sensors; determining whether the plurality of sensors are faulty, and generating sensor compensation information corresponding to the faulty sensor based on the sensor information of the valid sensors and the characteristic relationship; generating a second fusion result based on the sensor information of the valid sensors and the sensor compensation information;

[0008] According to this configuration and method, the present invention determines the characteristic relationship between the sensor information of a sensor that has not yet failed and the first fusion result at that time. When a sensor fails, sensor compensation information is generated based on the sensor information of the valid sensor and the characteristic relationship, achieving the effect of self-adaptive compensation. Therefore, the present invention can improve object detection function and stability, reduce discontinuities in output information, and achieve driving stability. It can also prevent the system from shutting down its autonomous driving function and ensure driving safety, stability, and comfort. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram of an embodiment of a multi-sensor self-adaptive fault compensation system of the present invention; [Figure 2] 1 is a top view of a plurality of sensors provided on a vehicle and their sensing ranges according to the present invention; [Figure 3A] 1 is a schematic diagram (1) of a waveform of a simulation result of the present invention. [Figure 3B] 10 is a schematic diagram (2) of the waveform of the simulation result of the present invention. [Figure 3C] 10 is a schematic diagram (3) of the waveform of the simulation result of the present invention. [Figure 4] 10 is a flowchart of a first embodiment showing the flow of generating a fault message by a processor in the present invention. [Figure 5] 1 is a flowchart showing a process for tracking and matching a target object according to the present invention; [Figure 6] 10 is a flowchart showing a flow of generating a fault message by a processor according to a second embodiment of the present invention. [Figure 7] 10 is a flowchart showing a flow of generating a fault message by a processor according to a third embodiment of the present invention. [Figure 8] 1 is a first flowchart showing the process of the processor performing self-adaptive compensation in the present invention; [Figure 9]10 is a flowchart (2) showing the flow of the processor performing self-adaptive compensation in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] As shown in FIGS. 1 and 2 , the multi-sensor self-adaptive fault compensation system of the present invention is applied to a vehicle V, which may be an autonomous vehicle. The multiple sensors 10 are installed at several locations on the vehicle V to sense the surroundings of the vehicle V. The storage 20 is installed on the vehicle V and may be, for example, a hard disk drive (HDD), a solid-state drive (SSD), a memory, a memory card, etc. The processor 30 may be installed in a vehicle control unit (VCU) or an electronic control unit (ECU) of the vehicle V and is electrically connected to the multiple sensors 10 and the storage 20. Each sensor 10 independently detects an object and outputs sensor information 100 to the processor 30. The sensor information 100 includes an object detection result, such as a vehicle ahead of the vehicle V (hereinafter referred to as a target object FV) and the relative distance between the target object FV and the vehicle V.

[0011] For example, to detect the situation ahead of the vehicle V, two or more of the multiple sensors 10, including a first corner radar sensor 11 (Corner Radar Sensor), a second corner radar sensor 12, a forward radar sensor 13, an optical radar sensor 14 (Light Detection And Ranging, LIDAR), and a forward-looking camera 15, are used, and the first corner radar sensor 11 and the second corner radar sensor 12 are provided on the left and right sides of the front of the vehicle V, respectively.

[0012] Since the multiple sensors 10 each have their own field of view, when installing the multiple sensors 10 on the vehicle V, the sensors 10 can be installed in appropriate positions as needed, and in this case, the field of view of any two of the multiple sensors 10 may overlap. A matching table is stored in the storage 20, and the matching table is preset with default data for defining the matching state of the multiple sensors 10. Any two matched sensors 10 have overlapping field of view. Taking FIG. 2 as an example, the field of view 110 of the first corner radar sensor 11, the field of view 130 of the forward radar sensor 13, the field of view 140 of the optical radar sensor 14, and the field of view 150 of the forward-view camera 15 overlap, and therefore are matched with each other. The detection range 120 of the second corner radar sensor 12, the detection range 130 of the forward radar sensor 13, the detection range 140 of the optical radar sensor 14, and the detection range 150 of the forward-looking camera 15 also overlap and are therefore matched with each other, but the detection ranges 110, 120 of the first corner radar sensor 11 and the second corner radar sensor 12 do not overlap and are therefore not matched with each other. The processor 30 reads the matching table from the storage 20 and refers to the table to determine the matching status of the multiple sensors 10.

[0013] Furthermore, the processor 30 can execute the programs of the fusion module 31, the fault judgment module 32, and the self-adaptive compensation module 33, and the programs of the fusion module 31, the fault judgment module 32, and the self-adaptive compensation module 33 are stored in the storage 20 so that they can be read and executed by the processor 30. As shown in FIG. 1, the fault judgment module 32 is connected to the multiple sensors 10 and the fusion module 31, and the self-adaptive compensation module 33 is connected to the multiple sensors 10, the fault judgment module 32, and the fusion module 31.

[0014] The processor 30 generates a first fusion result Z1 or a second fusion result Z2 using a fusion module 31, where the first fusion result Z1 is generated by the fusion module 31 based on sensor information 100 from multiple sensors 10 (i.e., when all of the multiple sensors 10 are valid), and the second fusion result Z2 is generated by the fusion module 31 based on sensor information 100 from partially valid sensors 10 (i.e., when at least one sensor has failed) and sensor compensation information C, and the information of the first fusion result Z1 and the information of the second fusion result Z2 each include information on the object detection result (i.e., the target object FV and the relative distance between the vehicle V and the target object FV). Note that such sensor fusion technology is common knowledge in the technical field to which it belongs, so a detailed description will not be provided.

[0015] Furthermore, the processor 30 can determine whether any one of the multiple sensors 10 has failed using the fault determination module 32. In the present invention, the processor 30 provides three embodiments for determining whether the multiple sensors 10 have failed, which will be described later. If the processor 30 determines through the fault determination module 32 that one of the multiple sensors 10 has failed, the fault determination module 32 sends a fault message S1 to the self-adaptive compensation module 33. This fault message S1 may be a fault flag. On the other hand, if the fault determination module 32 determines that all of the multiple sensors 10 are valid, the processor 30 sends the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault determination module 32, causes the fusion module 31 to generate a first fusion result Z1, and stores the first fusion result Z1 in the storage 20. Furthermore, the processor 30 determines the feature relationship of the sensor information 100 of each sensor 10 using the self-adaptive compensation module 33. The feature relationship includes (1) difference information of the object detection result between the sensor information 100 of each sensor 10 and the sensor information 100 of each other sensor 10, and (2) difference information of the object detection result between the sensor information 100 of each sensor 10 and the first fusion result Z1. Note that the feature relationship may further include difference information of the object detection result between the sensor information 100 of each sensor 10 and the second fusion result Z2.

[0016] For convenience of explanation, the sensors 10 that are faulty are defined as faulty sensors, and the sensors that are not faulty are defined as valid sensors. When the self-adaptive compensation module 33 receives the fault message S1, the processor 30 generates sensor compensation information C based on the sensor information 100 of the valid sensors and the feature relationships, where the sensor compensation information C includes the target object FV and the relative distance between the vehicle V and the target object FV. In this way, the sensor compensation information C can compensate for the missing sensor information of the faulty sensor according to the faulty sensor. Then, the self-adaptive compensation module 33 sends the sensor information 100 of the valid sensors and the sensor compensation information C to the fusion module 31. The processor 30 then causes the fusion module 31 to generate a second fusion result Z2 and store the second fusion result Z2 in the storage 20.

[0017] In this way, the processor 30 can perform control and judgment related to autonomous driving based on the first fusion result Z1 or the second fusion result Z2. That is, to stabilize the autonomous driving state and prevent the autonomous driving function from being affected or interrupted unexpectedly, the first fusion result Z1 is adopted when it is determined that all sensors 10 are valid, and the second fusion result Z2 is adopted when it is determined that at least one sensor 10 is faulty.

[0018] Simulation results (examples) of the present invention will be described below. In an embodiment in which the multiple sensors 10 include a forward-looking camera 15 and a forward radar sensor 13, the first waveform W1 shown in Fig. 3A is a waveform of the relative distance between the vehicle V and the target object FV contained in the sensor information 100 of the forward-looking camera 15. The second waveform W2 shown in Fig. 3A is a waveform of the relative distance between the vehicle V and the target object FV contained in the sensor information 100 of the forward radar sensor 13. The third waveform W3 shown in Fig. 3A is a waveform of the relative distance between the vehicle V and the target object FV contained in the information of the first fusion result Z1. Note that because the forward-looking camera 15 and the forward radar sensor 13 are installed in different locations on the vehicle V and use different principles for identifying objects, the relative distances indicated by the first waveform W1, second waveform W2, and third waveform W3 do not completely match, but the trends are consistent.

[0019] As described above, the processor 30 determines the characteristic relationship of the sensor information 100 of each sensor 10 through the self-adaptive compensation module 33. As shown in Fig. 3A, at any time tx, the characteristic relationship corresponding to the forward-viewing camera 15 includes a first relative distance difference Da and a second relative distance difference Db. The first relative distance difference Da is difference information (i.e., the difference value between the first waveform W1 and the second waveform W2 at time tx) between the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward-viewing camera 15 (i.e., the first waveform W1) and the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward radar sensor 13 (i.e., the second waveform W2). The second relative distance difference Db is difference information (i.e., the difference value between the first waveform W1 and the second waveform W2 at time tx) between the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward-viewing camera 15 (i.e., the first waveform W2). The feature relationship corresponding to the forward radar sensor 13 includes the first relative distance difference Da and the third relative distance difference Dc, and the third relative distance difference Dc refers to the difference information (i.e., the difference value between the third waveform W3 and the second waveform W2 at time tx) between the relative distance between the vehicle V and the target object FV included in the sensor information 100 of the forward radar sensor 13 (i.e., the second waveform W2) and the relative distance between the vehicle V and the target object FV included in the first fusion result Z1 (i.e., the third waveform W3). Similarly, the processor 30, via the self-adaptive compensation module 33, can also determine that the feature relationship corresponding to the first fusion result Z1 includes the second relative distance difference Db and the third relative distance difference Dc.

[0020] As shown in FIG. 3B, the first waveform W1 has an interruption section W1_loss between 12 seconds and 18 seconds, which simulates a state in which the forward-looking camera 15 is out of order during the interruption section W1_loss. It can be seen that the third waveform W3 approaches and overlaps with the second waveform W2 during the interruption section W1_loss. This result indicates that the first fusion result Z1 is almost entirely dependent on the sensor information 100 of the forward radar sensor 13.

[0021] As shown in Figure 3C, the self-adaptive compensation means of the present invention compensates for the interruption section W1_loss of the first waveform W1 shown in Figure 3B with the waveform of the sensor compensation information C, thereby maintaining the continuity of the first waveform W1. Also, the overall third waveform W3 shown in Figure 3C is closer to the third waveform W3 of Figure 3A, reflecting the compensation results of the sensor compensation information C.

[0022] The process of the multi-sensor self-adaptive fault compensation method of the present invention will be described in detail below.

[0023] 4, when the processor 30 receives sensor information 100 from the multiple sensors 10, it checks whether a first fusion result Z1 exists (step S01). For example, every time the fusion module 31 generates a first fusion result Z1, it generates a flag corresponding to this result, and the processor 30 checks whether the first fusion result Z1 exists based on this flag.

[0024] If the determination result in step S01 is "NO," the processor 30 proceeds to determine whether the multiple sensors 10 are frozen (step S02). For example, each sensor 10 transmits sensor information 100 to the processor 30, and the difference between the count values ​​of two pieces of sensor information 100 received by the processor 30 from each sensor 10 is typically not zero. Therefore, if the difference between the count values ​​of two pieces of sensor information 100 received by the processor 30 from a certain sensor 10 is zero, the processor 30 can determine that the sensor 10 is frozen. On the other hand, if the determination result in step S02 is "NO," this indicates that none of the multiple sensors 10 are frozen. In this case, the processor 30 transmits the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault determination module 32 (step S03), thereby causing the fusion module 31 to generate a first fusion result Z1.

[0025] A first embodiment in which the processor 30 generates a fault message S1 using the fault determination module 32 will be described below. For ease of understanding, in this embodiment, the multiple sensors 10 include a first sensor and a second sensor, which are matched to each other in a matching table in the storage 20. As shown in FIG. 4 , if the determination result in step S01 is “YES,” the processor 30 uses the fault determination module 32 to perform target object tracking matching for each of the sensor information 100 from the multiple sensors 10 against the first fusion result Z1 (step S04), thereby determining whether the sensor information 100 from the multiple sensors 10 and the first fusion result Z1 recognize the same target object FV. The target object tracking matching process shown in FIG. 5 is common knowledge in the technical field to which it pertains. Briefly, the processor 30 executes a program based on a calculation method known as a Kalman filter (Kalman Filter) to generate a first predicted fusion result based on the first fusion result Z1 (step S041). Next, a data association algorithm, such as a Hungarian Algorithm program, is executed to generate a matching result based on the first predicted fusion result and the sensor information of the first sensor (step S042). If this matching result is, for example, True or 1, it indicates that the matching result of the first sensor is successful, and at this time, the processor 30 can recognize the same target object FV from the sensor information of the first sensor and the first predicted fusion result. Conversely, if the matching result is, for example, False or 0, it indicates that the matching of the first sensor is unsuccessful, and the processor 30 cannot recognize the same target object FV from the sensor information of the first sensor and the first predicted fusion result. The second sensor is processed in the same way, and the aforementioned Kalman filter algorithm program and Hungarian algorithm program are stored in storage 20 so that they can be accessed and executed by the processor 30.

[0026] If matching of both the first and second sensors is successful, the processor 30 transmits the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault determination module 32, causes the fusion module 31 to generate a first fusion result for the next time point, and stores the first fusion result for the next time point in the storage 20. On the other hand, if matching of the first sensor fails, the processor 30 determines, via the fault determination module 32, whether the target object FV in the first prediction fusion result is located within the sensing range of the first sensor (S05). Furthermore, since the processor 30 generates spatial coordinate information around the vehicle V, the position of the vehicle V, the position of the target object FV, and the sensing ranges of the multiple sensors 10 can be identified using the coordinate information. Since these are common knowledge in the technical field to which they pertain, they will not be described in detail. In this way, the processor 30 determines, based on the coordinate information, whether the target object FV in the first prediction fusion result is located within the sensing range of the first sensor, and determines, via the fault determination module 32, whether the sensing range of the first sensor overlaps with the sensing range of the second sensor (step S05). In addition, the processor 30 can determine whether the sensing range of the first sensor overlaps with the sensing range of the second sensor based on the matching table stored in the storage 20 .

[0027] Also, if the judgment result of step S05 is "NO", it indicates that the target object FV in the first predicted fusion result is not present in the detection range of the first sensor, and / or the detection range of the first sensor does not overlap with the detection range of the second sensor. This may be due to the target object FV having moved outside the detection range of some sensors 10, so the processor 30 continues to send the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault judgment module 32 (step S03), causing the fusion module 31 to generate the first fusion result at the next point in time.

[0028] On the other hand, if the judgment result of step S05 is "YES", it means that the target object FV in the first prediction fusion result is present in the detection range of the first sensor and the second sensor, and the detection range of the first sensor overlaps with the detection range of the second sensor. In other words, if the sensor information of the second sensor includes the target object FV and the sensor information of the first sensor does not include the target object FV, the processor 30 determines the first sensor to be a faulty sensor and the second sensor to be a valid sensor through the fault judgment module 32, and the fault judgment module 32 sends a fault message S1 to the self-adaptive compensation module 33 (step S06).

[0029] A second embodiment in which the processor 30 generates a fault message S1 using the fault determination module 32 will be described below. As shown in FIG. 6 , the sensor information 100 output from each sensor 10 to the processor 30 may include credibility information, which may be, for example, true or false, indicating whether the object detection result is credible. In this manner, the processor 30 uses the fault determination module 32 to determine whether the credibility information of each of the sensors 10 is credible (step S07). If the determination result in step S07 is "NO," this indicates that some of the sensor information 100 from the sensors 10 lacks credibility. In this case, as described in step S05 above, the processor 30 uses the fault determination module 32 to determine each unreliable sensor 10 as a faulty sensor and each credible sensor 10 as a valid sensor, and the fault determination module 32 sends a fault message S1 to the self-adaptive compensation module 33. On the other hand, if the judgment result of step S07 is "YES", it means that the sensor information 100 of the multiple sensors 10 is all reliable, so the processor 30 sends the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault judgment module 32 (step S03), and causes the fusion module 31 to generate the first fusion result at the next time point.

[0030] A third embodiment will now be described in which the processor 30 generates a fault message S1 using the fault determination module 32. As shown in FIG. 7, the third embodiment is a combination of the first and second embodiments. If the determination result in step S01 is "YES," the processor 30 further determines whether the authenticity information of each of the multiple sensors 10 is authentic using the fault determination module 32 (step S07'). On the other hand, if the determination result in step S07' is "NO," the processor 30 sends the fault message S1 to the self-adaptive compensation module 33 via the fault determination module 32 (step S06'). Subsequently, if the determination result in step S07' is "YES," i.e., if the information is authentic, the processor 30 performs target object tracking and matching of the sensor information 100 of each of the multiple sensors 10 with the first fusion result Z1 using the fault determination module 32 (step S04'). As in the previous embodiment, the multiple sensors include a first sensor and a second sensor, and if the first sensor and the second sensor are both successfully matched, the processor 30 sends the sensor information 100 of the multiple sensors 10 to the fusion module 31 via the fault judgment module 32 (step S03), causing the fusion module 31 to generate a first fusion result at the next time point; on the other hand, if the matching of the first sensor fails, the processor 30 executes step S05 as described above via the fault judgment module 32.

[0031] The above describes three examples in which the processor 30 generates the fault message S1 using the fault judgment module 32. Hereinafter, we will further explain the processing performed by the self-adaptive compensation module 33 of the processor 30 when the fault message S1 has not been received and after the fault message S1 has been received.

[0032] 8, the processor 30 determines, via the self-adaptive compensation module 33, whether or not a fault message S1 has been received from the fault determination module 32 (step S11). If the determination result of step S11 is “NO,” this indicates that the processor 30 has not detected a faulty sensor 10 via the fault determination module 32. The processor 30 then determines, via the self-adaptive compensation module 33, whether or not a first fusion result Z1 exists (step S12; see step S01 above). If the determination result of step S12 is “YES,” the processor 30 extracts the feature relationship based on the first fusion result Z1 and the sensor information 100 of the multiple sensors 10 (step S13) and updates the feature relationship information (step S14). On the other hand, if the determination result of step S12 is “NO,” the processor 30 reads default feature relationship information from the storage 20 via the self-adaptive compensation module 33 (step S15) and updates the default feature relationship information. The purpose of these steps S13 to S15 is to track the target object FV traveling in front of the vehicle V in real time using the updated feature relationship information.

[0033] 9, if the determination result in step S11 is "YES," the processor 30 determines, via the self-adaptive compensation module 33, whether all of the matched sensors 10 are faulty (step S16). As described above, the processor 30 determines whether both the first sensor and the second sensor are faulty. If the determination result in step S16 is "YES," this indicates that the target object FV may have moved out of the sensing ranges of both the first sensor and the second sensor, but does not indicate that both the first sensor and the second sensor have simultaneously failed. In this case, the processor 30 transmits the current sensor information 100 of the multiple sensors 10 to the fusion module 31 via the self-adaptive compensation module 33 (step S17), causing the fusion module 31 to generate a first fusion result for the next time point.

[0034] On the other hand, if the judgment result of step S16 is "NO," it indicates that only some of the sensors 10 can detect the target object FV. That is, as described above, only the second sensor (valid sensor) detects the target object FV, and the first sensor (faulty sensor) does not detect the target object FV. In this case, the processor 30 further determines whether a first fusion result from a previous time point exists (step S18). Here, the processor 30 may store the first fusion result previously generated via the fusion module 31 in the storage 20 and define it as the first fusion result from the previous time point.

[0035] If the determination result in step S18 is "YES," the processor 30 causes the self-adaptive compensation module 33 to perform prediction based on the first fusion result at a previous time point and generate prediction fusion result information (step S19). The processor 30 also determines, through the self-adaptive compensation module 33, that the feature relationship corresponding to the prediction fusion result includes the target object FV and the predicted relative distance between the target object FV and the vehicle V. At this time, the processor 30 performs a Kalman filter algorithm to perform prediction based on the first fusion result at a previous time point and generate the prediction fusion result. Note that the feature relationship includes the feature relationship between the prediction fusion result and the sensor information 100 when the multiple sensors 10 are not faulty. Therefore, the processor 30 causes the self-adaptive compensation module 33 to generate sensor compensation information C based on the sensor information 100 of the available sensors and the feature relationship (step S20).

[0036] Continuing from the previous example, if neither the forward camera 15 nor the forward radar sensor 13 is faulty, the first relative distance difference Da is 1.3 meters, the second relative distance difference Db is 1.1 meters, and the third relative distance difference Dc is 0.2 meters. If, at a later point in time, the forward camera 15 fails and becomes the faulty sensor but the forward radar sensor 13 is valid, the processor 30 generates information on the prediction fusion result. If this prediction fusion result corresponds to the predicted waveform W3_predict shown in FIG. 3C , that is, if the forward camera 15 fails, the third relative distance difference will be the predicted difference information (e.g., 0.3 meters) between the relative distance between the vehicle V and the target object FV (i.e., the second waveform W2) included in the sensor information 100 of the forward radar sensor 13 (the valid sensor) and the predicted relative distance between the vehicle V and the target object FV (i.e., the predicted waveform W3_predict) included in the prediction fusion result. If the difference value (0.3 meters) of this predicted information is added to the second relative distance difference Db (1.1 meters) when both the forward camera 15 and the forward radar sensor 13 are not faulty, the first relative distance difference when the forward camera 15 fails is predicted to be 1.4 meters (i.e., 0.3 + 1.1 = 1.4), and sensor compensation information C is generated based on this predicted first relative distance.

[0037] If the judgment result of step S18 is "NO", the processor 30 reads the information of the default fusion result from the storage 20 using the self-adaptive compensation module 33. At this time, the feature relationship as described above is the feature relationship between the default fusion result and the sensor information 100 when the multiple sensors 10 are not faulty, and the processor 30 generates the sensor compensation information C based on the sensor information 100 of the valid sensors and this feature relationship using the self-adaptive compensation module 33.

[0038] After generating the sensor compensation information C, the processor 30 further determines, via the self-adaptive compensation module 33, whether there is missing sensor information (step S21). For example, as shown in FIG. 3B, if the processor 30 determines that the first waveform W1 is a discontinuous waveform due to the existence of an interruption section W1_loss within the first waveform W1, it determines that there is missing sensor information. If the determination result in step S21 is "NO," this indicates that the missing sensor information has been compensated for by the sensor compensation information C. Also, in FIG. 3C, the first waveform W1 already includes the sensor compensation information C, i.e., the interruption section W1_loss has been replaced with the sensor compensation information C. In this case, the processor 30 continues to send the current sensor information 100 of the multiple sensors 10 to the fusion module 31 via the self-adaptive compensation module 33 (step S22). At the same time, the self-adaptive compensation module 33 extracts feature relationships based on the second fusion result Z2 (step S23), and updates the feature relationship information using the feature relationships extracted from the second fusion result Z2 (step S14).

[0039] As described above, the present invention has the following effects.

[0040] 1. In response to the tracking and matching of the target object in step S04, the processor 30 tracks the target object FV based on the prediction of the first fusion result Z1, and determines whether each sensor 10 has lost track of the target object FV, and further determines whether a sensor has malfunctioned.

[0041] Second, the processor 30 determines the characteristic relationship (distance) that exists between the first fusion result at the previous time point, the current first fusion result Z1, and the current sensor information 100 of each sensor 10. In the event of a sensor failure, the processor 30 uses this characteristic relationship to automatically generate the missing sensor data, i.e., sensor compensation information C, in a self-adaptive manner and provides it to the fusion module 31, thereby avoiding problems such as deviation in the target detection relative distance (discontinuity in output) and inability to detect.

[0042] Third, the processor 30 improves the stability of object detection and tracking through the cooperative operation of the fusion module 31, the fault judgment module 32, and the self-adaptive compensation module 33. [Explanation of symbols]

[0043] 10 sensors 11 First corner radar sensor 12 Second corner radar sensor 13 Forward radar sensor 14 Optical Radar Sensor 15 Forward-looking camera 20. Storage 30 processors 31 Fusion Module 32 Fault Judgment Module 33 Self-adaptive compensation module 100 Sensor Information 110, 120, 130, 140, 150 detection range V vehicle FV Target Object Z1 first fusion result Z2 second fusion result C Sensor Compensation Information S1 Fault message W1 1st waveform W1_loss Interrupted section W2 Second waveform W3 third waveform W3_predict Predicted waveform Da first relative distance difference Db Second relative distance difference Dc Third relative distance difference At the time of tx

Claims

1. 1. A multi-sensor self-adaptive fault compensation system, comprising: A plurality of sensors each outputting sensor information; a processor electrically connected to the plurality of sensors; The processor: a fusion module that generates a first fusion result or a second fusion result; a fault determination module connecting the plurality of sensors and the fusion module, transmitting sensor information of the plurality of sensors to the fusion module, and causing the fusion module to generate the first fusion result; a self-adaptive compensation module connected to the plurality of sensors, the fault determination module, and the fusion module, for determining a feature relationship between sensor information of the plurality of sensors; When the fault determination module determines that one of the plurality of sensors is faulty, it sends a fault message to the self-adaptive compensation module, and defines the faulty sensors of the plurality of sensors as faulty sensors and the non-faulty sensors as valid sensors; When the self-adaptive compensation module receives the fault message, it generates sensor compensation information corresponding to the faulty sensor according to the sensor information of the valid sensor and the characteristic relationship; The self-adaptive compensation module transmits the sensor information of the available sensors and the sensor compensation information to the fusion module, and causes the fusion module to generate the second fusion result.

2. the plurality of sensors includes a first sensor and a second sensor; The fault judgment module performs tracking and matching of the target object with the sensor information of the plurality of sensors against the first fusion result, respectively; If the first sensor and the second sensor are successfully matched, the fault judgment module sends the sensor information of the plurality of sensors to the fusion module, causing the fusion module to generate a first fusion result at a next time point; if the matching of the first sensor is unsuccessful, the fault judgment module determines whether the object in the first predicted fusion result is located within the sensing range of the first sensor, and determines whether the sensing range of the first sensor overlaps with the sensing range of the second sensor; If the answer is "NO," the fault judgment module sends the sensor information of the plurality of sensors to the fusion module, and causes the fusion module to generate a first fusion result at the next time point; On the other hand, if the answer is "YES," the fault judgment module determines that the first sensor is the faulty sensor and sends the fault message to the self-adaptive compensation module.

3. The sensor information output by each of the sensors includes authenticity information; The fault determination module determines whether the authenticity information of each of the plurality of sensors is authentic; If the answer is "NO", the fault determination module determines the unreliable sensor as the fault sensor and sends the fault message to the self-adaptive compensation module; On the other hand, if the answer is "YES," the fault judgment module sends the sensor information of the plurality of sensors to the fusion module, and causes the fusion module to generate a first fusion result at the next time point.

4. The sensor information output by each of the sensors includes authenticity information; The fault determination module determines whether the authenticity information of each of the plurality of sensors is authentic; If the answer is "NO", the fault determination module determines the unreliable sensor as the fault sensor and sends the fault message to the self-adaptive compensation module; On the other hand, if the answer is “YES,” the fault determination module performs tracking and matching of the target object with the sensor information of each of the plurality of sensors against the first fusion result, and the plurality of sensors includes a first sensor and a second sensor; If the first sensor and the second sensor are successfully matched, the fault judgment module sends the sensor information of the plurality of sensors to the fusion module, causing the fusion module to generate a first fusion result at a next time point; if the matching of the first sensor is unsuccessful, the fault judgment module determines whether the object in the first predicted fusion result is located within the sensing range of the first sensor, and determines whether the sensing range of the first sensor overlaps with the sensing range of the second sensor; If the answer is "NO," the fault judgment module sends the sensor information of the plurality of sensors to the fusion module, and causes the fusion module to generate a first fusion result at the next time point; On the other hand, if the answer is "YES," the fault judgment module determines that the first sensor is the faulty sensor and sends the fault message to the self-adaptive compensation module.

5. When the self-adaptive compensation module receives the fault message, it determines whether there is a first fusion result from an earlier time point; If the answer is 'YES', the self-adaptive compensation module performs prediction based on the first fusion result at a previous time point to generate predicted fusion result information, and the feature relationship is a feature relationship between the predicted fusion result information and sensor information of the plurality of sensors; On the other hand, if the answer is "NO," the self-adaptive compensation module reads information of a default fusion result from storage, and the feature relationship is a feature relationship between the default fusion result and the sensor information of the multiple sensors.

6. 1. A method performed by a processor electrically connected to a plurality of sensors, comprising: generating a first fusion result based on sensor information of the plurality of sensors; determining a feature relationship between the sensor information of the plurality of sensors; determining whether the plurality of sensors are faulty, and generating sensor compensation information corresponding to the faulty sensor based on the sensor information of the valid sensors and the characteristic relationship; generating a second fusion result based on the sensor information of the available sensors and the sensor compensation information;

7. the plurality of sensors includes a first sensor and a second sensor; In the step of determining whether the plurality of sensors are malfunctioning, Track and match the sensor information of the plurality of sensors with the first fusion result, respectively, to the target object; If the first sensor and the second sensor are successfully matched, a first fusion result for a next time point is generated based on the sensor information of the plurality of sensors; if the first sensor is unsuccessful in matching, it is determined whether the target object in the first predicted fusion result is located within a sensing range of the first sensor, and whether the sensing range of the first sensor overlaps with that of the second sensor; If the answer is 'NO', generate a first fusion result at the next time point based on the sensor information of the plurality of sensors; On the other hand, if the answer is "YES," the method of claim 6 further comprises determining the first sensor as the faulty sensor.

8. The sensor information output by each of the sensors includes authenticity information; In the step of determining whether the plurality of sensors are malfunctioning, determining whether the authenticity information of each of the plurality of sensors is authentic; If the answer is "NO," the unreliable sensor is determined to be the faulty sensor. On the other hand, if the answer is 'YES', generating a first fusion result for a next time point based on the sensor information of the plurality of sensors.

9. The sensor information output by each of the sensors includes authenticity information; In the step of determining whether the plurality of sensors are malfunctioning, determining whether the authenticity information of each of the plurality of sensors is authentic; If the answer is "NO," the unreliable sensor is determined to be the faulty sensor. On the other hand, if the answer is "YES," the sensor information of each of the plurality of sensors is subjected to tracking and matching of the target object with the first fusion result, and the plurality of sensors includes a first sensor and a second sensor; If the first sensor and the second sensor are successfully matched, a first fusion result for a next time point is generated based on the sensor information of the plurality of sensors. On the other hand, if the matching of the first sensor is unsuccessful, it is determined whether the target object in the first predicted fusion result is located within a sensing range of the first sensor, and whether the sensing range of the first sensor overlaps with that of the second sensor. If the answer is 'NO', generate a first fusion result at the next time point based on the sensor information of the plurality of sensors; On the other hand, if the answer is "YES," determining the first sensor as the faulty sensor.

10. When it is determined that one of the plurality of sensors has failed, it is determined whether or not a first fusion result from an earlier point in time exists; If the answer is 'YES', prediction is performed based on the first fusion result at a previous time point to generate information on a predicted fusion result, and the feature relationship is a feature relationship between the predicted fusion result and sensor information of the plurality of sensors; On the other hand, if the answer is "NO," information of a default fusion result is read, and the feature relationship is a feature relationship between the default fusion result and the sensor information of the plurality of sensors.

Images