Image input / output system failure handling method, device, equipment, and medium

Abstract

An embodiment of the present disclosure discloses a method, apparatus, device, and medium for handling a fault in an image input / output system, the method including: acquiring fault information of the image input / output system; determining a target type and target level to which the current fault belongs based on the fault information; and performing a fault recovery process for the current fault corresponding to the target type in response to the target level being a first preset level. The embodiment of the present disclosure enables a lower layer to perform recovery process for a fault that can be quickly recovered from by itself, without exposing it to an upper layer application for autonomous driving, thereby avoiding the downgrade of autonomous driving functions performed by the upper layer application and eliminating the need for the user to immediately take over the vehicle, thereby reducing erroneous downgrade of autonomous driving functions and helping to improve the user experience.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims priority to a Chinese patent application filed on December 19, 2022, bearing application number 202211637384.9 and entitled "Method and apparatus, device and medium for handling failures in image input / output systems," the entire contents of which are incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to a fault handling technique, and more particularly to a method, apparatus, device and medium for handling faults in an image input / output system. [Background technology]

[0003] In the field of autonomous driving, autonomous driving behavior decision-making is one of the key technologies for realizing autonomous driving functions, and autonomous driving behavior decision-making refers to an autonomous vehicle sensing surrounding environmental information through sensors, comprehensively considering the surrounding environment, obstacles, vehicle merging and priority rules, etc., and matching it with the experiential knowledge in the autonomous driving library to decide on driving behavior appropriate for the current environment. During the autonomous driving process, different types of failures may occur, such as sensor hardware failures and software failures, which may require the vehicle to downgrade its autonomous driving functions, thereby worsening the user's driving experience. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present disclosure provide a method, apparatus, device, and medium for handling image input / output system failures, which can help reduce the occurrence of downgrading of autonomous driving functions, thereby improving user experience. [Means for solving the problem]

[0005] According to one aspect of an embodiment of the present disclosure, there is provided a method for handling an image input / output system failure, including the steps of: acquiring failure information of an image input / output system; determining a target type and target level to which a current failure belongs based on the failure information; and, in response to the target level being a first preset level, performing a failure recovery process for the current failure corresponding to the target type.

[0006] According to another aspect of an embodiment of the present disclosure, there is provided an image input / output system fault processing device, including: a first acquisition module for acquiring fault information of the image input / output system; a first determination module for determining a target type and target level to which the current fault belongs based on the fault information; and a first processing module for performing fault recovery processing corresponding to the target type on the current fault in response to the target level being a first preset level.

[0007] According to a further aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program for executing the method for handling an image input / output system fault described in any of the above embodiments of the present disclosure.

[0008] According to yet another aspect of an embodiment of the present disclosure, there is provided an electronic device including a processor and a memory for storing commands executable by the processor, the processor being used to read and execute the executable commands from the memory to realize the method for handling an image input / output system failure described in any of the above embodiments of the present disclosure. [Effects of the Invention]

[0009] Based on the image input / output system fault handling method, device, equipment, and medium provided in the above embodiments of the present disclosure, faults in the image input / output system in autonomous driving are classified into levels. For faults that occur in lower-layer software and that can be recovered by the lower layer itself, such as minor faults, general faults, and single major faults, recovery processing can be performed directly in the lower layer, without exposing it to the upper-layer autonomous driving application, and avoiding the upper-layer application from downgrading the autonomous driving function as a result, and eliminating the need for the user to immediately take over the vehicle, which helps to reduce erroneous downgrading of the autonomous driving function and helps to improve the user experience. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an exemplary application scenario of the image input / output system fault handling method provided by the present disclosure; [Figure 2] 1 is a flowchart of a method for handling a fault in an image input / output system provided by one exemplary embodiment of the present disclosure. [Figure 3] 10 is a flowchart of a method for handling a fault in an image input / output system provided by another exemplary embodiment of the present disclosure. [Figure 4] 10 is a flowchart of a method for handling a fault in an image input / output system provided by a further exemplary embodiment of the present disclosure; [Figure 5] 1 is a schematic structural diagram of an image input / output system fault processing device provided by one exemplary embodiment of the present disclosure; [Figure 6] FIG. 10 is a schematic structural diagram of an image input / output system fault processing device provided by another exemplary embodiment of the present disclosure; [Figure 7] 5 is a schematic structural diagram of a first asynchronous fault processing module 507 provided by one exemplary embodiment of the present disclosure; FIG. [Figure 8] FIG. 10 is a schematic structural diagram of an apparatus provided according to another exemplary embodiment of the present disclosure. [Figure 9]1 is a schematic structural diagram of an application example of an electronic device of the present disclosure; DETAILED DESCRIPTION OF THE INVENTION

[0011] In order to explain the present disclosure, exemplary embodiments of the present disclosure will be described in detail below with reference to the drawings. Obviously, it should be understood that the described embodiments are only some embodiments of the present disclosure, not all embodiments, and the present disclosure is not limited to the exemplary embodiments.

[0012] It should be noted that the relative arrangement of parts and steps, formulas and numerical values ​​described in these examples do not limit the scope of the present disclosure unless specifically stated otherwise.

[0013] Summary of the Disclosure In the process of realizing this disclosure, the inventors discovered that in the field of autonomous driving, autonomous driving behavior decision-making is one of the key technologies for realizing autonomous driving functions, and autonomous driving behavior decision-making refers to an autonomous vehicle sensing surrounding environmental information through sensors, comprehensively considering the surrounding environment, obstacles, vehicle merging and priority rules, etc., and matching it with the experiential knowledge in the autonomous driving library to decide on driving behavior appropriate for the current environment. During the autonomous driving process, different types of failures such as hardware failures and software failures may occur, which will reduce the driving safety of the vehicle and thereby worsen the user's driving experience.

[0014] Illustrative Overview FIG. 1 is an exemplary application scenario of the image input / output system fault handling method provided by the present disclosure.

[0015] In an autonomous driving scenario, raw data collected by an image sensor can be transmitted to a video input / output (VIO) system. The VIO system can perform certain processing on the raw data to obtain processed image data, which can be transmitted to an upper-layer application. The upper-layer application can then perform environment sensing based on the image data for use in autonomous driving behavior decision-making and control. The VIO system fault processing method disclosed herein (executed in an VIO system fault processing device) can be used to monitor the VIO system, obtain fault information for the VIO system, and determine the target type and target level to which the current fault belongs based on the fault information. The target type can include various faults that occur in the image transmission or processing path of the VIO system and physical hardware faults (e.g., image sensor disconnection). The various faults that occur in the image transmission or processing path can include, for example, transmission process verification failure faults, image transmission protocol (MIPI) physical faults, image abnormality faults, transmitted image size mismatch faults, memory write failures, data acquisition failure faults, data frame loss faults, etc., and can be specifically configured according to actual needs. The target level may be set according to actual needs, for example, the target level may include a minor level, a general level, a single serious level, a fatal level, etc. After determining the target type and target level to which the current fault belongs, a fault recovery process corresponding to the target type can be performed for the current fault in response to the target level being a first preset level, where the first preset level refers to a level at which a lower layer can recover itself, and may be set according to actual needs, for example, the first preset level may be a minor level, and if the target type is a certain type of fault within the software, an internal recovery can be performed for the current fault, without needing to report it to an upper layer application, thereby avoiding the upper layer application from downgrading the autonomous driving function, and without the user having to immediately take over the vehicle, which helps to reduce erroneous downgrading of the autonomous driving function and helps to improve the user experience.

[0016] Exemplary Methods 2 is a flowchart of an image input / output system fault handling method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms, and includes the following steps, as shown in FIG.

[0017] In step 201, fault information of the image input / output system is acquired.

[0018] Here, a video input / output (VIO) system (which may be abbreviated to system) is a system for processing raw data collected by an image sensor to obtain image data, and the VIO system may include a hardware portion and a software portion. The hardware portion may include a hardware unit for performing image processing, such as an ISP (Image Signal Processing) unit. The software portion may include, but is not limited to, software for controlling the hardware unit to complete image processing. The fault information of the VIO system may include fault-related information such as a fault type, a fault level, a time when the fault occurred, an image link where the fault occurred, fault description information, etc., and may be specifically set according to actual needs. The fault types may include various faults occurring in the image transmission or processing path of the image input / output system and physical hardware faults (e.g., a fatal fault such as an image sensor disconnection). The various faults occurring in the image transmission or processing path may include, for example, a transmission process verification failure fault, a physical image transmission protocol (MIPI) failure, an image abnormality fault, a transmitted image size mismatch fault, a memory write failure, a data acquisition failure fault, a data frame loss fault, etc., and may be specifically set according to actual needs. The fault level may be set according to actual needs, and may, for example, include a minor level, a general level, a single serious level, a fatal level, etc. The fault information may be acquired in any feasible manner. For example, various fault detection and reporting functions may be set in the system, and when a fault is detected, an appropriate fault may be reported to the device, and specifically set according to actual needs.

[0019] In one alternative example, step 201 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first acquisition module executed by the processor.

[0020] In step 202, the target type and target level to which the current fault belongs are determined based on the fault information.

[0021] Here, the correspondence between the failure information and the failure type and failure level can be determined and stored in advance based on the failure conditions that may actually occur, and after the failure information is acquired, the target type and target level to which the current failure belongs can be determined based on the failure information and the correspondence.

[0022] In one alternative example, step 202 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first determination module executed by the processor.

[0023] In step 203, in response to the target level being the first preset level, a fault recovery process corresponding to the target type is performed for the current fault.

[0024] Here, the first preset level refers to a level at which the lower layer can recover itself and can be set according to actual needs. For example, the first preset level can include at least one of a minor level, a general level, and a single-critical level. If the target level is determined to be the first preset level, it indicates that the current failure may be a self-recoverable failure, and a failure recovery process corresponding to the target type can be performed on the current failure. For example, if the target type is a certain type of failure within the lower layer software that can be automatically recovered by recovering the software, the lower layer can recover from the failure by simply controlling the automatic recovery of the software, and there is no need to report it to an upper layer application.

[0025] In one alternative example, step 203 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first processing module executed by the processor.

[0026] The image input / output system fault handling method provided in this embodiment classifies faults in the image input / output system in autonomous driving into levels, and for first preset level faults that can be recovered by the lower layer itself, such as minor faults, general faults, and single major faults that occur in the lower layer software, the lower layer can directly perform recovery processing, without exposing it to the upper layer application of autonomous driving, and avoiding the upper layer application from downgrading the autonomous driving function as a result, and eliminating the need for the user to immediately take over the vehicle, which helps to reduce erroneous downgrading of the autonomous driving function and helps to improve the user experience.

[0027] FIG. 3 is a flowchart of a method for handling a fault in an image input / output system provided by another exemplary embodiment of the present disclosure.

[0028] In one alternative example, the method of the embodiment of the present disclosure may further include the following steps:

[0029] In step 204, fault monitoring information of the image input / output system is acquired.

[0030] Here, the fault monitoring information refers to information obtained by monitoring, recording, or stating fault conditions that occur in the image input / output system. The fault monitoring information includes, for example, fault condition information, fault type, fault level, etc. of each fault that has already occurred and been monitored, but is not limited to these. The fault condition information may include information on changes in the fault condition for each fault type, and the fault condition may include two states: fault occurrence and fault removal. Taking a type of fault as an example, the information on changes in the fault condition for that fault type may be represented, for example, by the time periods during which different fault conditions of that fault type have been recorded. That is, the information on changes in the fault condition for that fault type may include the time periods during which the fault condition is in a fault occurrence state and the time periods during which the fault condition is in a fault removal state. Based on the fault condition information, the fault occurrence condition within a certain period of time, for example, the frequency of occurrence of any fault, may be determined.

[0031] In one alternative example, step 204 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a fault monitoring module executed by the processor.

[0032] In step 205, it is determined whether the asynchronous event trigger condition is met based on the fault monitoring information.

[0033] Here, the asynchronous event trigger condition can be set according to actual needs. The asynchronous event trigger condition is used to determine whether an asynchronous event needs to be generated. The generation of an asynchronous event indicates that a fault condition that cannot be recovered from by the lower layer has occurred and needs to be reported to an upper layer application through an asynchronous event. For example, the asynchronous event trigger condition may include the monitoring of a fatal fault, multiple faults occurring frequently within a short period of time, frequent stability faults in the system, and failure of a fault recovery process. A fatal fault may be a physical fault, such as a camera disconnection. Multiple faults occurring frequently within a short period of time refers to a significant problem occurring in the entire system, resulting in a large number of faults occurring within a short period of time. A frequent stability fault in the system refers to a stability problem occurring in the system, and the same type of fault is continuously monitored. A failure of a fault recovery process refers to a failure in a lower layer's own fault recovery process for a fault originally belonging to the first preset level to successfully complete the fault recovery, resulting in the fault not being resolved. Whether the asynchronous event trigger condition is met can be determined by matching the fault monitoring information with the asynchronous event trigger condition.

[0034] In one alternative example, step 205 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a second decision module executed by the processor.

[0035] In step 206, in response to the fault monitoring information satisfying the asynchronous event trigger condition, a first asynchronous event corresponding to the fault monitoring information is generated.

[0036] Here, if the fault monitoring information is different, the asynchronous event that needs to be generated will also be different, and specifically, it can be set according to actual needs. For example, if the fault monitoring information is that a single fatal fault has been monitored, the first asynchronous event is a single fatal fault event; if the fault monitoring information is that multiple faults that occur frequently within a short period of time have been monitored, the corresponding first asynchronous event is a frequent fault event within a short period of time; and if the fault monitoring information is that frequent stability faults have been monitored in the system, the corresponding first asynchronous event is a stability fault event.

[0037] In one alternative example, step 206 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first generating module executed by the processor.

[0038] In step 207, fault handling is performed based on the first asynchronous event.

[0039] Here, different fault handling methods can be set for different first asynchronous events to solve corresponding fault problems. For example, if the first asynchronous event is a single fatal fault event, fault handling corresponding to the single fatal fault event is performed. For example, if the single fatal fault is a physical fault (e.g., camera disconnection) and a user needs to be notified about the handling, the fault notification information can be output. If the fault type corresponding to the single fatal fault is a software fault, the corresponding software can be restarted. However, the fault handling method is not limited to this.

[0040] In one alternative example, step 207 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first asynchronous fault processing module executed by the processor.

[0041] An embodiment of the present disclosure monitors for fault conditions occurring in an image input / output system. When a condition that the lower layer cannot recover from is detected, the condition can be reported to an upper layer application in real time and quickly through an asynchronous event, allowing the upper layer application to quickly handle the fault and resolve the fault problem in a timely manner, ensuring the driving safety of the vehicle and not affecting the normal operation of the autonomous driving function. This helps to improve the accuracy of the downgrade operation and further improve the user experience.

[0042] FIG. 4 is a flowchart of a method for handling an image input / output system failure provided by a further exemplary embodiment of the present disclosure.

[0043] In one optional example, step 207 of performing fault processing based on the first asynchronous event includes step 2071 of controlling to output fault presentation information in response to determining that the first asynchronous event is a single fatal fault event and that the fault type corresponding to the first asynchronous event is a physical fault.

[0044] Here, the single fatal failure event may correspond to a single physical failure (e.g., disconnection of an image sensor) or a single software failure (e.g., VIO link software failure). Different failure handling methods may be adopted for different single fatal failure events. For the first asynchronous event whose failure type is a physical failure, it indicates that the software cannot currently recover automatically and physical recovery is required. Therefore, a diagnostic report may be issued and the user may be prompted to recover manually.

[0045] In one alternative example, step 2071 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a first processing unit executed by the processor.

[0046] In step 2072, in response to determining that the first asynchronous event is a single fatal failure event and that the failure type corresponding to the first asynchronous event is a software failure, control is performed to restart the image link of the image input / output system.

[0047] Here, the image link (abbreviated as VIO link) of the image input / output system is a processing link for image signal processing in the image input / output system, and raw data collected by image sensors (cameras) at each viewpoint of the vehicle can be processed by the image input / output system to obtain image data required for different subsequent functions. Each viewpoint can correspond to at least one VIO link, and each VIO link can complete image processing of the raw data from the image sensor at the corresponding viewpoint to obtain the corresponding image data. In the case of a single fatal failure event, which is a software failure, a failure situation that cannot be quickly recovered from currently occurs in the VIO link, indicating that image data cannot be obtained, and failure recovery can be performed through operations such as restarting (resetting). Therefore, failure recovery can be achieved by controlling the image link of the image input / output system to restart.

[0048] In one alternative example, step 2072 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a second processing unit executed by the processor.

[0049] In step 2073, in response to the first asynchronous event being a frequent failure event within a short period of time or a stability failure event, it is determined whether image data can be acquired within a preset time.

[0050] Here, the preset time can be set according to actual needs, and can include, for example, the current time and a certain period of time before the current time, but is not limited to this. Whether image data can be acquired can be determined by monitoring the image data acquisition result of the upper layer application. For example, the upper layer application can be set to return the acquisition result to the device of the embodiment of the present disclosure every time it acquires image data, or the device of the embodiment of the present disclosure can request the acquisition result from the upper layer application. Specifically, the preset time can be set according to actual needs.

[0051] In one alternative example, step 2073 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a third processing unit executed by the processor.

[0052] In step 2074, in response to being able to acquire image data within the preset time, the normal operating state of the image input / output system is maintained.

[0053] Here, if image data can be acquired within the preset time, it indicates that the current asynchronous event mechanism has been triggered, but the current system has already recovered to normal, or there is just an accidental unstable situation in the system. If a function related to autonomous driving (e.g., an autonomous driving behavior decision-making function) is currently operating, the normal operating state of the image input / output system can be temporarily maintained to ensure that the function related to autonomous driving continues to operate.

[0054] In one alternative example, step 2074 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a fourth processing unit executed by the processor.

[0055] In step 2075, in response to the inability to acquire image data within the preset time, control is performed to restart the image link of the image input / output system.

[0056] Here, if image data cannot be acquired within a preset time, it indicates that a fault condition has occurred in the current system that cannot be quickly recovered from by itself, and the fault can be recovered through operations such as restarting. Therefore, controlling the image link of the image input / output system to restart is useful for solving the fault problem.

[0057] In one alternative example, step 2075 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a fifth processing unit executed by the processor.

[0058] In the embodiments of the present disclosure, for different asynchronous events, the faults can be resolved through corresponding fault handling methods, without affecting the autonomous driving function, and the normal operation of the image input / output system can be maintained so that the autonomous driving function can operate normally, thereby avoiding function downgrade, which helps to improve the accuracy of the downgrade operation and further improve the user experience.

[0059] In one optional example, after step 2074 of maintaining the normal operating state of the image input / output system in response to being able to acquire image data within a preset time, the method of an embodiment of the present disclosure may further include step 301 of controlling the image input / output system to restart in response to the completion of a first target function involving the image input / output system.

[0060] Here, the first target function may be any function that requires image data to be acquired from the image input / output system during autonomous driving, such as an autonomous driving behavior decision-making function, but is not limited to this. If it is determined that image data can be successfully acquired within a preset time after a first asynchronous event, which is a short-term frequent failure event or a stability failure event of the system, is triggered, the normal operating state of the image input / output system can be maintained and the image input / output system can be prevented from being restarted, so that the first target function involving the image input / output system can operate normally without being downgraded. Restarting the image input / output system after the first target function is completed can be controlled without affecting the first target function. At this time, the image input / output system can be controlled to be restarted, which helps to avoid recurrence of a failure corresponding to the previously triggered asynchronous event (including a short-term frequent failure event or a stability failure event), allows the image input / output system to operate safely thereafter, and further improves vehicle driving safety.

[0061] In one alternative example, step 301 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a sixth processing unit executed by the processor.

[0062] In one optional example, after step 207 of performing fault processing based on the first asynchronous event, the method further includes step 208 of determining, in response to the processing result of the fault processing being a failure, a target image link in which a fault currently exists based on the fault information corresponding to the first asynchronous event.

[0063] Here, if the processing result of the fault processing is a failure, it indicates that an irrecoverable fault has occurred in the target image link where the fault currently exists, and the target image link where the fault currently exists can be determined based on the image link where the fault has occurred, which is included in the fault information corresponding to the first asynchronous event.

[0064] In one alternative example, step 208 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a second processing module executed by the processor.

[0065] In step 209, in response to the target image link having a pre-placed alternative image link and the alternative image link being in a non-faulty state, environmental information of the viewpoint corresponding to the target image link is determined based on image data acquired by the alternative image link and a substitution algorithm and / or a substitution model corresponding to the pre-placed alternative image link.

[0066] Here, an alternative image link corresponding to each image link can be pre-configured. For example, an image link corresponding to a forward-view camera can be replaced by image links corresponding to the left and right front-view cameras, respectively. That is, image data within the forward-view field of view range covered by the combination of the left and right front-view images is used to determine image data within the forward-view field of view range. This allows each autonomous driving function that requires forward-view image data to continue functioning without being downgraded. The specific alternative image link can be configured according to actual needs. The status of the alternative image link can include two states: a non-fault state and a fault state. In actual applications, the status of the specific alternative image link can be determined based on the fault state of each image link, which is maintained in real time. For example, the status of each image link can be maintained in real time based on the aforementioned fault information and fault monitoring information, and can be specifically configured according to actual needs. The alternative algorithm and / or alternative model corresponding to the alternative image link refers to a sensing algorithm or sensing model that performs environment sensing based on image data acquired by the alternative image link, and can be specifically set according to actual needs. The sensing model can include, but is not limited to, a target detection model obtained by pre-training, a semantic segmentation model, etc. Based on the corresponding alternative algorithm and / or alternative model, environmental information of the viewpoint corresponding to the target image link can be extracted from the image data acquired by the alternative image link, and the function of the target image link can be completed based on the alternative image link and the corresponding algorithm model.

[0067] In one alternative example, step 209 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a third processing module executed by the processor.

[0068] In the embodiments of the present disclosure, for any target image link, after the failure handling of the upper layer application fails, the function of the target image link can be replaced by an alternative image link, so that the autonomous driving function involving the target image link can continue to operate without the need for downgrading, which helps to further improve the accuracy of the downgrading operation and further improve the user experience.

[0069] In one alternative example, when the alternative image link is running, there is indeed a fault in the target image link that cannot be recovered, and the effect achieved by the alternative solution may have a certain difference from the effect achieved by the target image link. Therefore, when the alternative image link is used, fault warning information can be controlled to be output to notify the user of the current system problem.

[0070] In one alternative example, different levels of control over the automated driving function can be exercised depending on the alternative effect level of the alternative image link, for example, allowing the user to partially take over driving and retain some of the automated driving function, which can be specifically set according to actual needs.

[0071] In one optional example, the method of an embodiment of the present disclosure may further include step 210 of controlling the second target function involving the target image link to be downgraded and outputting downgrade presentation information in response to the target image link not having a pre-placed alternative image link or the pre-placed alternative image link being in a faulty state.

[0072] Here, the second target function is a function in the automatic driving function that involves a target image link, for example, an automatic driving behavior decision-making function. If the target image link does not have a pre-arranged alternative image link or the pre-arranged alternative image link is in a faulty state, it can indicate that the function replacement has failed. In this case, in order to enable the vehicle to operate safely, the second target function that involves the target image link can be controlled to be downgraded, and downgrade notification information can be output, thereby prompting the user to take over driving in a timely manner and helping to avoid traffic accidents.

[0073] In one alternative example, step 210 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a fourth processing module executed by the processor.

[0074] The embodiments of the present disclosure can downgrade an autonomous driving function after a situation in which the autonomous driving function can continue to operate is eliminated, which helps reduce the occurrence of false trigger situations for downgrade operations and thereby helps improve the user experience.

[0075] In one optional example, after step 203 of performing a fault recovery process corresponding to the target type for the current fault in response to the target level being the first preset level, the method of an embodiment of the present disclosure may further include the following steps:

[0076] In step 401, in response to the processing result of the failure recovery processing being a failure, a second asynchronous event is generated.

[0077] Here, the second asynchronous event may be a failure recovery process failure event, and may indicate that the processing result of the failure recovery process is a failure if recovery cannot be performed for the current failure of the first preset level. The second asynchronous event is similar to the first asynchronous event and is used to trigger failure processing of an upper layer application, and detailed description thereof will be omitted.

[0078] In one alternative example, step 401 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a second generating module executed by the processor.

[0079] In step 402, fault handling is performed based on the second asynchronous event.

[0080] Here, if the current fault cannot be recovered by itself, the fault can be handled by controlling it through an upper layer application, for example, the image input / output system can be controlled to be restarted, and specifically, it can be set according to actual needs.

[0081] In the embodiment of the present disclosure, when a first preset level fault cannot be recovered by itself, the fault can be reported to an upper layer application in real time and quickly through an asynchronous event mechanism, so that the upper layer application can quickly handle the fault and resolve the fault problem in a timely and fast manner.

[0082] In one alternative example, step 402 may be performed by the processor invoking a corresponding command stored in memory, or may be performed by a second asynchronous fault processing module executed by the processor.

[0083] In one optional example, step 402 of performing failure processing based on the second asynchronous event includes controlling an image link of the image input / output system to be restarted in response to the second asynchronous event being a failure recovery processing failure event.

[0084] Here, for a fault that cannot be recovered by itself, the upper layer application can resolve the current fault by controlling the image link of the image input / output system to restart, thereby quickly restoring the image link to normal function and allowing the automatic driving function to operate normally.

[0085] In one optional example, an asynchronous fault handling application (also referred to as an upper layer asynchronous fault handling application or an upper layer asynchronous fault handling module) can be configured in the upper layer application, which is used to quickly perform corresponding fault handling in the upper layer in response to asynchronous events, and the asynchronous events can include first asynchronous events and second asynchronous events, and can also include other asynchronous events configured according to actual needs.

[0086] The embodiments of the present disclosure can realize a fault collection and monitoring function for lower-layer system software by classifying faults into levels and types, which helps to block the reporting of most faults that can be quickly recovered from themselves. Furthermore, based on the collected fault monitoring information, an asynchronous event mechanism can be used to realize asynchronous reporting of fault situations that cannot be recovered from themselves, which can realize real-time and fast handling of asynchronous event faults. Furthermore, if the fault handling fails or the fault fails to be automatically recovered from, function replacement can be performed through an alternative image link, which allows the autonomous driving function to continue operating, helps to reduce the downgrade rate of the autonomous driving function, and helps to improve the user experience.

[0087] Each embodiment and each optional example of the present disclosure can be implemented alone, and if there is no conflict, they can also be combined and implemented in any combination, and can be specifically set according to actual needs.

[0088] Any of the image input / output system fault handling methods provided by the embodiments of the present disclosure may be executed by any device with appropriate data processing capabilities, including, but not limited to, a terminal device, a server, etc. Alternatively, any of the image input / output system fault handling methods provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor invokes a corresponding command stored in a memory to execute any of the image input / output system fault handling methods described in the embodiments of the present disclosure. Hereinafter, the description will be omitted.

[0089] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by hardware associated with program commands, and the above-mentioned program can be stored in a computer-readable storage medium, which performs the steps of the above-mentioned method embodiments when the program is executed, and the above-mentioned storage medium includes various media that can store program code, such as a ROM, a RAM, a magnetic disk, or an optical disk.

[0090] Exemplary Apparatus 5 is a schematic structural diagram of an image input / output system fault processing device provided by an exemplary embodiment of the present disclosure. The device of this embodiment can implement a corresponding method embodiment of the present disclosure. For example, the device shown in FIG. 5 includes a first acquisition module 501, a first determination module 502, and a first processing module 503.

[0091] The first acquisition module 501 is used to acquire fault information of the image input / output system, the first determination module 502 is used to determine the target type and target level to which the current fault belongs based on the fault information, and the first processing module 503 is used to perform fault recovery processing corresponding to the target type for the current fault in response to the target level being a first preset level.

[0092] FIG. 6 is a schematic structural diagram of an image input / output system fault processing device provided by another exemplary embodiment of the present disclosure.

[0093] In one optional example, the apparatus of the embodiment of the present disclosure further includes: a fault monitoring module 504 used to acquire fault monitoring information of the image input / output system; a second determination module 505 used to determine whether an asynchronous event trigger condition is met based on the fault monitoring information; a first generation module 506 used to generate a first asynchronous event corresponding to the fault monitoring information in response to the fault monitoring information satisfying the asynchronous event trigger condition and report it to a first asynchronous fault processing module; and a first asynchronous fault processing module 507 used to perform fault processing based on the first asynchronous event.

[0094] FIG. 7 is a schematic structural diagram of the first asynchronous fault processing module 507 provided by one exemplary embodiment of the present disclosure.

[0095] In one optional example, the first asynchronous fault processing module 507 includes: a first processing unit 5071 used to control the output of fault display information in response to determining that the first asynchronous event is a single fatal fault event and that the fault type corresponding to the first asynchronous event is a physical fault; a second processing unit 5072 used to control the restart of an image link of the image input / output system in response to determining that the first asynchronous event is a single fatal fault event and that the fault type corresponding to the first asynchronous event is a software fault; a third processing unit 5073 used to determine whether image data can be acquired within a preset time in response to the first asynchronous event being a frequent failure event within a short period of time or a stability failure event; a fourth processing unit 5074 used to maintain a normal operating state of the image input / output system in response to the image data being acquired within the preset time; and a fifth processing unit 5075 used to control the restart of an image link of the image input / output system in response to the image data not being acquired within the preset time.

[0096] In one optional example, the device of the embodiment of the present disclosure further includes a sixth processing unit 5076 used to control the image input / output system to restart in response to the completion of a first target function involving the image input / output system.

[0097] In one optional example, the apparatus of the embodiment of the present disclosure further includes a second processing module 508 used to determine a target image link in which a fault currently exists based on fault information corresponding to the first asynchronous event in response to the processing result of the fault processing being a failure, and a third processing module 509 used to determine environmental information of a viewpoint corresponding to the target image link based on image data acquired by the alternative image link and an alternative algorithm and / or alternative model corresponding to the pre-arranged alternative image link in response to the target image link having a pre-arranged alternative image link and the alternative image link being in a non-fault state.

[0098] In one optional example, the device of the embodiment of the present disclosure further includes a fourth processing module 601 used to control the second target function related to the target image link to be downgraded and output downgrade presentation information in response to the target image link not having a pre-placed alternative image link or the pre-placed alternative image link being in a faulty state.

[0099] In one optional example, the apparatus of the embodiment of the present disclosure further includes a second generation module 602 used to generate a second asynchronous event in response to the processing result of the failure recovery processing being a failure, and a second asynchronous failure processing module 603 used to perform failure processing based on the second asynchronous event.

[0100] Here, the second asynchronous fault processing module 603 may be the same module as the first asynchronous fault processing module 507 described above, or may be two independent modules, which can be specifically configured according to actual needs.

[0101] In one alternative example, the second asynchronous fault processing module 603 is specifically used to control the restart of the image link of the image input / output system in response to the second asynchronous event being a fault recovery processing failure event.

[0102] In one alternative example, FIG. 8 is a schematic framework diagram of an autonomous driving behavior decision-making system provided according to an exemplary embodiment of the present disclosure. In this example, the autonomous driving behavior decision-making system may include an upper layer application, a hardware abstraction layer, a hardware driver layer, and a hardware portion. The image input / output system fault processing device of the embodiment of the present disclosure is configured in the hardware abstraction layer and the upper layer application to complete the image input / output system fault processing method of the embodiment of the present disclosure. Here, the device may include device portion 1 and device portion 2. Device portion 1 may include the above-mentioned first acquisition module, first determination module, first processing module, fault monitoring module, second determination module, first generation module, second generation module, etc., and device portion 1 is configured in the hardware abstraction layer. Device portion 2 may include the above-mentioned first asynchronous fault processing module and second asynchronous fault processing module, and device portion 2 is configured in the upper layer application. The upper layer application may further include various upper layer applications for autonomous driving functions, such as a sensing application that senses the environment based on image data, in addition to the device portion 2 of the embodiment of the present disclosure, and a detailed description thereof will be omitted. The hardware may include an image sensor, an ISP unit for performing signal processing on raw data collected by the image sensor, and various hardware processing units for performing image processing on image data.The hardware driver layer includes software programs that drive each piece of hardware. The hardware abstraction layer is a software library related to the commonalities of each hardware driver abstracted based on the hardware driver layer. Libcam represents a software library related to the image sensor (camera). LIBVIO represents a software library related to the VIO system. Each module of the device part 1 monitors, records, and compiles statistics on failures related to the image sensor and VIO system hardware and software. For failures that can be quickly recovered from by the lower layer, the device part 1 can control or indirectly control the quick recovery of the lower layer. For monitored failures that cannot be recovered from by the lower layer, asynchronous events (including the above-mentioned first and second asynchronous events) can be generated in real time and reported to the device part 2 of the upper layer application through an asynchronous mechanism. In response to each asynchronous event, each module of the device part 2 can quickly and real-timely handle the corresponding asynchronous failure in the upper layer, for example, by controlling the VIO system to restart or controlling the replacement of a failed target image link with an alternative image link, thereby enabling the autonomous driving function to operate normally. If the fault handling fails and it is determined that function replacement is not possible, or if there is another fault that cannot be resolved, a diagnosis report can be made, and the corresponding autonomous driving function can be controlled to be downgraded, and the user can be notified. For specific handling of various situations, please refer to the above content, and the description will be omitted here.

[0103] The beneficial technical effects corresponding to the exemplary embodiments of the present apparatus may refer to the corresponding beneficial technical effects of the exemplary method portion described above, and will not be described here.

[0104] Exemplary Electronic Devices 9 is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0105] The processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or command execution functionality, and may control other components within the electronic device 10 to perform desired functions.

[0106] The memory 12 may include one or more computer program products, including various forms of computer-readable storage media, such as, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program commands may be stored in the computer-readable storage medium, and the processor 11 may execute the one or more computer program commands to implement the methods of each embodiment of the present disclosure described above and / or other desired functions.

[0107] In one example, electronic device 10 may further include input devices 13 and output devices 14, with these components connected to one another via a bus system and / or other form of connection (not shown).

[0108] The input device 13 may further include, for example, a keyboard, a mouse, and the like.

[0109] The output device 14 can output various types of information to the outside, and can include, for example, a display, a speaker, a printer, a communication network, and a remote output device connected thereto.

[0110] Of course, for the sake of simplicity, Fig. 9 shows only some of the components related to the present disclosure in the electronic device 10, and omits components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may further include any other appropriate components according to specific application situations.

[0111] Exemplary Computer Program Products and Computer-Readable Storage Media In addition to the methods and apparatus described above, embodiments of the present disclosure may also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the method steps of various embodiments of the present disclosure described in the "Exemplary Methods" section above.

[0112] The computer program product may have program code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and common procedural programming languages ​​such as "C" or similar programming languages, for carrying out operations of embodiments of the present disclosure. The program code may run entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0113] Additionally, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the method steps of various embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0114] The computer-readable storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a readable signal medium or a readable storage medium. The computer-readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0115] Although the basic principles of the present disclosure have been described above with reference to specific embodiments, the benefits, advantages, effects, etc. mentioned in the present disclosure are merely illustrative and not limiting, and it is not believed that these benefits, advantages, effects, etc. must be included in each embodiment of the present disclosure. Furthermore, the specific details disclosed above merely serve to illustrate and facilitate understanding, and are not limiting, and the details do not limit the scope of the present disclosure to those specific details.

[0116] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure also intends to include these modifications and variations.

Claims

1. acquiring failure information of the image input / output system; determining a target type and a target level to which the current fault belongs based on the fault information; and in response to the target level being a first preset level, performing a fault recovery process for the current fault corresponding to the target type.

2. acquiring fault monitoring information of the image input / output system; determining whether an asynchronous event trigger condition is met based on the fault monitoring information; generating a first asynchronous event corresponding to the fault monitoring information in response to the fault monitoring information satisfying the asynchronous event trigger condition; The method of claim 1 , further comprising: performing fault handling based on the first asynchronous event.

3. The step of performing fault handling based on the first asynchronous event includes: controlling to output fault presentation information in response to determining that the first asynchronous event is a single fatal fault event and that the fault type corresponding to the first asynchronous event is a physical fault; In response to determining that the first asynchronous event is a single fatal failure event and that the failure type corresponding to the first asynchronous event is a software failure, controlling an image link of the image input / output system to be restarted; determining whether image data can be acquired within a preset time in response to the first asynchronous event being a frequent failure event within a short period of time or a stability failure event; maintaining a normal operating state of the image input / output system in response to the image data being acquired within the preset time; 3. The method of claim 2, further comprising the step of: controlling an image link of said image input / output system to be restarted in response to said image data not being acquired within said preset time.

4. 4. The method of claim 3, wherein after the step of maintaining the image input / output system in a normal operating state in response to the image data being able to be acquired within the preset time, the method further includes the step of controlling the image input / output system to restart in response to a first target function involving the image input / output system being completed.

5. after the step of performing failure processing based on the first asynchronous event, determining a target image link in which a fault currently exists based on the fault information corresponding to the first asynchronous event in response to a processing result of the fault processing being a failure; 3. The method of claim 2, further comprising: in response to the target image link having a pre-placed alternative image link and the alternative image link being in a non-faulty state, determining environmental information of a viewpoint corresponding to the target image link based on image data acquired by the alternative image link and an alternative algorithm and / or alternative model corresponding to the pre-placed alternative image link.

6. 6. The method of claim 5, further comprising the step of controlling a second target function involving the target image link to be downgraded and outputting downgrade presentation information in response to the target image link not having the alternative image link pre-placed or the pre-placed alternative image link being in a fault state.

7. After the step of performing a fault recovery process corresponding to the target type for the current fault in response to the target level being a first preset level, the method further comprises: generating a second asynchronous event in response to a processing result of the failure recovery processing being a failure; The method of claim 1 , further comprising: performing fault handling based on the second asynchronous event.

8. 8. The method according to claim 7, wherein the step of performing failure processing based on the second asynchronous event includes a step of controlling to restart an image link of the image input / output system in response to the second asynchronous event being a failure recovery processing failure event.

9. a first acquisition module for acquiring failure information of the image input / output system; a first determination module for determining a target type and a target level to which the current fault belongs based on the fault information; a first processing module for performing a fault recovery process corresponding to the target type for the current fault in response to the target level being a first preset level.

10. A computer-readable storage medium storing a computer program for executing the method for handling a fault in an image input / output system according to any one of claims 1 to 8.

11. a processor; a memory for storing instructions executable by said processor; The electronic device is used to implement the method for handling a failure in an image input / output system according to any one of claims 1 to 8, wherein the processor reads and executes the executable command from the memory.

Images