Method, device and medium for processing image input / output system failure

Abstract

Embodiments of the present disclosure disclose a method, device, equipment and medium for processing fault of image input and output system, wherein the method comprises: obtaining fault information of the image input and output system; determining a target type and a target level to which the current fault belongs based on the fault information; and performing fault recovery processing corresponding to the target type on the current fault in response to the target level being a first preset level. For the fault that can be quickly self-recovered at the bottom layer, the recovery processing is directly performed at the bottom layer, without being exposed to the upper application of the autonomous driving, and without the need for the user to perceive, thereby effectively reducing the false degradation of the autonomous driving function, improving the user experience, and solving the technical problems of the prior art, such as the poor user driving experience caused by the direct reporting of the autonomous driving fault triggering the degradation of the autonomous driving function.

Description

Technical Field

[0001] This disclosure relates to fault handling techniques, and in particular to a method, apparatus, device, and medium for handling faults in an image input / output system. Background Technology

[0002] In the field of autonomous driving, autonomous driving behavior decision-making is one of the key technologies for realizing autonomous driving functions. Autonomous driving behavior decision-making refers to the process by which autonomous vehicles perceive information about their surrounding environment through sensors, comprehensively consider rules regarding the surrounding environment, obstacles, vehicle merging, and yielding, and match this information with experiential knowledge in the autonomous driving database to determine the appropriate driving behavior for the current environment. During autonomous driving, various types of faults may occur, such as sensor hardware failures and software failures. In related technologies, when a fault occurs, it is usually directly reported to the upper-layer application. After detecting the fault, the upper-layer application immediately degrades the autonomous driving function, making it unusable and requiring the user to take over the vehicle immediately, resulting in a poor user driving experience. Summary of the Invention

[0003] To address the technical issues such as the poor user driving experience caused by direct reporting of autonomous driving malfunctions triggering degraded autonomous driving functions, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, device, and medium for handling malfunctions in an image input / output system.

[0004] According to one aspect of the present disclosure, a method for handling faults in an image input / output system is provided, comprising: acquiring fault information of the image input / output system; determining, based on the fault information, the target type and target level of the current fault; and, in response to the target level being a first preset level, performing fault recovery processing on the current fault corresponding to the target type.

[0005] According to another aspect of the present disclosure, an image input / output system fault processing apparatus is provided, comprising: a first acquisition module, configured to acquire fault information of the image input / output system; a first determination module, configured to determine the target type and target level of the current fault based on the fault information; and a first processing module, configured to perform fault recovery processing on the current fault corresponding to the target type in response to the target level being a first preset level.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the image input / output system fault handling method described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image input / output system fault handling method described in any of the above embodiments of the present disclosure.

[0008] Based on the image input / output system fault handling method, apparatus, device, and medium provided in the above embodiments of this disclosure, by classifying the faults of the image input / output system in autonomous driving into levels, for minor faults, general faults, and single serious faults that occur in the underlying software and can be self-recovered at the underlying level, the recovery processing can be performed directly at the underlying level without exposing it to the upper-layer applications of autonomous driving, and without requiring the user to perceive it. This can effectively reduce the false degradation of autonomous driving functions, improve the user experience, and solve the technical problems of existing technologies where direct reporting of autonomous driving faults triggers the degradation of autonomous driving functions, resulting in a poor user driving experience.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is an exemplary application scenario of the image input / output system fault handling method provided in this disclosure;

[0012] Figure 2 This is a flowchart illustrating a method for handling image input / output system malfunctions provided in an exemplary embodiment of this disclosure;

[0013] Figure 3 This is a flowchart illustrating a method for handling image input / output system malfunctions provided in another exemplary embodiment of this disclosure;

[0014] Figure 4 This is a flowchart illustrating a method for handling image input / output system malfunctions provided in yet another exemplary embodiment of this disclosure;

[0015] Figure 5 This is a schematic diagram of the structure of an image input / output system fault processing apparatus provided in an exemplary embodiment of the present disclosure;

[0016] Figure 6This is a schematic diagram of the structure of an image input / output system fault processing apparatus provided in another exemplary embodiment of the present disclosure;

[0017] Figure 7 This is a schematic diagram of the structure of the first asynchronous fault handling module 507 provided in an exemplary embodiment of this disclosure;

[0018] Figure 8 This is a schematic diagram of the structure of an apparatus provided in another exemplary embodiment of this disclosure;

[0019] Figure 9 This is a schematic diagram of the structure of one application embodiment of the electronic device disclosed herein. Detailed Implementation

[0020] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0021] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0023] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0025] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0026] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0027] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0028] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0032] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0033] This disclosure outlines

[0034] In realizing this disclosure, the inventors discovered that in the field of autonomous driving, autonomous driving behavior decision-making is one of the key technologies for realizing autonomous driving functions. Autonomous driving behavior decision-making refers to the autonomous vehicle perceiving surrounding environmental information through sensors, comprehensively considering rules such as the surrounding environment, obstacles, vehicle merging, and yielding, and matching them with experience knowledge in the autonomous driving database to decide on driving behavior suitable for the current environment. During the autonomous driving process, different types of faults may occur, such as hardware faults and software faults. In related technologies, when a fault occurs, it is usually directly reported to the upper-layer application. After the upper-layer application detects the fault, it will immediately degrade the autonomous driving function, making the autonomous driving function directly unusable, requiring the user to take over the vehicle immediately, resulting in a poor user driving experience.

[0035] Exemplary Overview

[0036] Figure 1 This is an exemplary application scenario of the image input / output system fault handling method provided in this disclosure.

[0037] In autonomous driving scenarios, raw data collected by image sensors is transmitted to a Video Input / Output (VIO) system. The VIO system processes the raw data to obtain processed image data, which is then transmitted to upper-layer applications. These applications use the image data for environmental perception, decision-making, and control in autonomous driving. This disclosed method for handling VIO system faults (executed within a VIO fault handling device) allows for monitoring of the VIO system, acquisition of fault information, and determination of the target type and level of the fault. Target types can include various faults in the image transmission or processing path of the VIO system, as well as hardware physical faults (such as image sensor disconnection). Examples of faults in the image transmission or processing path include transmission verification failures, physical faults in the image transmission protocol (MIPI), image anomalies, mismatched image sizes, memory write failures, data acquisition failures, and data frame loss, etc., which can be set according to actual needs. Target levels can also be set according to actual needs, such as minor, moderate, single severe, and critical levels, etc. After determining the target type and target level of the current fault, in response to the target level being the first preset level, fault recovery processing corresponding to the target type is performed on the current fault. The first preset level refers to the level that can be recovered automatically at the lower level. It can be set according to actual needs. For example, if the first preset level is a minor level and the target type is a certain fault inside the software, then the current fault can be recovered internally without reporting to the upper layer application or letting the user perceive it. This can effectively reduce the false downgrade of autonomous driving functions, improve the user experience, and solve the technical problems of existing technologies where direct reporting of autonomous driving faults triggers the downgrade of autonomous driving functions, resulting in a poor driving experience for users.

[0038] Exemplary methods

[0039] Figure 2 This is a schematic flowchart illustrating a method for handling image input / output system faults according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 2 As shown, it includes the following steps:

[0040] Step 201: Obtain fault information of the image input / output system.

[0041] The Video Input / Output (VIO) system (or simply the system) is used to process the raw data acquired by the image sensor to obtain image data. It can include hardware components (hardware units for image processing, such as an ISP (Image Signal Processing) unit) and software components (software controlling the hardware units to complete image processing), with no specific limitations. Fault information for the VIO system can include fault type, fault level, fault occurrence time, the image link where the fault occurred, fault description, and other fault-related information, which can be set according to actual needs. Fault types can include various faults in the image transmission or processing path of the VIO system, and hardware physical faults (also known as fatal faults, such as image sensor disconnection). Various faults in the image transmission or processing path can include transmission process verification failures, image transmission protocol (MIPI) physical faults, image anomaly faults, transmitted image size mismatch faults, memory write faults, data acquisition failures, data frame loss faults, etc., which can be set according to actual needs. Fault levels can be set according to actual needs, such as minor, general, single severe, fatal, etc. Fault information can be obtained in any feasible way. For example, the system can be equipped with various fault detection and reporting functions. When a fault is detected, the corresponding fault is reported to the device. The specific settings can be configured according to actual needs.

[0042] Step 202: Based on the fault information, determine the target type and target level to which the current fault belongs.

[0043] In this system, the correspondence between fault information and fault type and fault level can be determined and stored in advance based on the actual possible fault situations. When the fault information is obtained, the target type and target level of the current fault can be determined based on the fault information and the correspondence.

[0044] Step 203: In response to the target level being the first preset level, perform fault recovery processing corresponding to the target type for the current fault.

[0045] The first preset level refers to the level that can be automatically recovered at the underlying level. It can be set according to actual needs. For example, the first preset level can include at least one of the following: minor level, general level, and single severe level. When the target level is determined to be the first preset level, it means that the current fault may be a self-recoverable fault. Then, fault recovery processing corresponding to the target type is performed on the current fault. For example, if the target type is a certain fault inside the underlying software that can be automatically recovered by recovery software, then fault recovery can be performed automatically at the underlying control software level without reporting to the upper-level application.

[0046] The image input / output system fault handling method provided in this embodiment classifies the faults of the image input / output system in autonomous driving into levels. For faults of the first preset level, such as minor faults, general faults, and single serious faults that occur in the underlying software, which can be self-recovered at the underlying level, the recovery processing is performed directly at the underlying level without exposing them to the upper-level applications of autonomous driving or allowing the user to perceive them. This can effectively reduce the false degradation of autonomous driving functions, improve the user experience, and solve the technical problems of existing technologies where direct reporting of autonomous driving faults triggers the degradation of autonomous driving functions, resulting in a poor driving experience for users.

[0047] Figure 3 This is a flowchart illustrating a method for handling image input / output system malfunctions provided in another exemplary embodiment of this disclosure.

[0048] In one optional example, the method disclosed herein further includes the following steps:

[0049] Step 204: Obtain fault monitoring information of the image input / output system.

[0050] Among them, fault monitoring information refers to the information obtained by monitoring, recording or statistically analyzing the faults that occur in the image input and output system. Fault monitoring information can include the fault status information, fault type, fault level, etc. of each fault that has occurred. Fault status information can include the change information of the fault status of each fault type. The fault status can include two states: fault occurrence and fault clearance. Taking a fault type as an example, the change information of the fault status of this fault type can be represented by the time of recording different fault states of this fault type. That is, the change information of the fault status of this fault type can include the time of each fault state being the fault occurrence state and the time of each fault state being the fault clearance state. Based on the fault status information, the occurrence of faults within a certain period of time can be determined, such as the occurrence frequency of any kind of fault.

[0051] Step 205: Based on the fault monitoring information, determine whether the asynchronous event triggering conditions are met.

[0052] The asynchronous event triggering conditions can be set according to actual needs. These conditions determine whether an asynchronous event needs to be generated. Generating an asynchronous event indicates that a fault has occurred that the underlying system cannot recover from autonomously, requiring reporting to the upper-layer application via an asynchronous event. For example, asynchronous event triggering conditions can include detecting a fatal fault, multiple faults occurring frequently in a short period, frequent system stability faults, and failure of fault recovery processing. A fatal fault could be a physical fault, such as a camera disconnection. Multiple faults occurring frequently in a short period refer to a serious system-wide problem causing a large number of faults to occur within a short time. Frequent system stability faults indicate system instability leading to the continuous detection of the same type of fault. Failure of fault recovery processing means that a fault that originally belonged to the first preset level failed to complete fault recovery during underlying autonomous fault recovery processing, leaving the fault unresolved. The asynchronous event triggering conditions are matched with fault monitoring information to determine whether they are met.

[0053] Step 206: In response to the fault monitoring information meeting the asynchronous event triggering conditions, generate the first asynchronous event corresponding to the fault monitoring information.

[0054] Different fault monitoring information will generate different asynchronous events, which can be set according to actual needs. For example, if the fault monitoring information detects a single fatal fault, the first asynchronous event will be the single fatal fault event; if the fault monitoring information detects multiple faults that occur frequently in a short period of time, the corresponding first asynchronous event will be the short-term frequent fault event; if the fault monitoring information detects frequent stability faults in the system, the corresponding first asynchronous event will be the stability fault event; and so on.

[0055] Step 207: Perform fault handling based on the first asynchronous event.

[0056] Different fault handling methods can be set for different first asynchronous events to resolve corresponding fault problems. For example, if the first asynchronous event is a single fatal fault event, then the fault handling corresponding to the single fatal fault event will be performed. For example, if the single fatal fault is a physical fault (such as camera disconnection), and the user needs to be prompted to handle it, then the output of fault prompt information will be controlled. If the fault type corresponding to the single fatal fault is a software fault, the corresponding software can be restarted, and the specifics are not limited.

[0057] This disclosure monitors faults occurring in the image input / output system. When a situation is detected that the underlying layer cannot recover autonomously, it can be reported to the upper-layer application in real time via asynchronous events. The upper-layer application can then quickly handle the fault, resolve the problem promptly, ensure vehicle driving safety, and not affect the normal operation of the autonomous driving function. This improves the accuracy of the downgrade operation and further enhances the user experience.

[0058] Figure 4 This is a flowchart illustrating a method for handling image input / output system malfunctions provided in another exemplary embodiment of this disclosure.

[0059] In an optional example, step 207, based on the first asynchronous event, performs fault handling, including:

[0060] Step 2071: In response to the first asynchronous event being a single fatal fault event, and determining that the fault type corresponding to the first asynchronous event is a physical fault, control output fault prompt information.

[0061] Among them, a single fatal failure event can correspond to a single physical failure (such as an image sensor disconnection) or a single software failure (such as a VIO link software failure). Different failure handling methods can be adopted for different single fatal failure events. For the first asynchronous event with a failure type of physical failure, it means that the current software cannot recover automatically and physical recovery is required. Therefore, it is necessary to perform diagnosis and reporting and prompt the user to manually recover.

[0062] Step 2072: In response to the first asynchronous event being a single fatal fault event, and determining that the fault type corresponding to the first asynchronous event is a software fault, control the restart of the image link of the image input / output system.

[0063] The image input / output system's image link (VIO link) is the processing link for image signal processing. Raw data collected by image sensors (cameras) from various viewpoints on the vehicle needs to be processed by the image input / output system to obtain the image data required for subsequent functions. Each viewpoint corresponds to at least one VIO link, and each VIO link is responsible for processing the raw data from its corresponding viewpoint's image sensor to obtain the corresponding image data. A single fatal software failure indicates a non-recoverable fault in the current VIO link, resulting in the inability to obtain image data. Fault recovery requires operations such as a reset. Therefore, fault recovery is achieved by controlling the restart of the image input / output system's image link.

[0064] Step 2073: In response to the first asynchronous event being a short-term frequent fault event or a stability fault event, determine whether image data can be obtained within a preset time.

[0065] The preset time can be set according to actual needs, such as including the current time and a certain amount of time preceding the current time, without any specific limitation. Whether image data can be obtained can be determined by monitoring the image data acquisition results of the upper-layer application. For example, the upper-layer application can be set to return the acquisition result to the device disclosed herein after each acquisition of image data, or the device disclosed herein can request the acquisition result from the upper-layer application when needed, which can be set according to actual needs.

[0066] Step 2074: In response to the ability to obtain image data within a preset time, maintain the normal working state of the image input / output system.

[0067] If image data can be obtained within a preset time, it means that although the asynchronous event mechanism has been triggered, the system has returned to normal or the system is only experiencing occasional instability. If the relevant functions of autonomous driving (such as autonomous driving behavior decision-making function) are currently running, the normal working state of the image input and output system can be temporarily maintained to ensure the continued operation of the relevant functions of autonomous driving.

[0068] Step 2075: In response to the inability to obtain image data within a preset time, control the restart of the image link of the image input / output system.

[0069] If image data cannot be obtained within a preset time, it indicates that the current system has experienced a fault that cannot be quickly self-recovered and requires a restart or other operations to recover from the fault. Therefore, the image link of the image input and output system is controlled to restart in order to resolve the fault.

[0070] This disclosure addresses different asynchronous events by providing corresponding fault handling methods to resolve faults. Without affecting the autonomous driving function, it maintains the normal operation of the image input / output system, ensuring the normal operation of the autonomous driving function without requiring function degradation, thus improving the accuracy of degradation operations and further enhancing the user experience.

[0071] In an optional example, after the image data is obtained within a preset time in response to step 2074, and the normal operation of the image input / output system is maintained, the method of this disclosure further includes:

[0072] Step 301: In response to the exit of the first target function in which the image input / output system participates, control the restart of the image input / output system.

[0073] The first target function can be any function in autonomous driving that requires acquiring image data from the image input / output system, such as autonomous driving behavior decision-making functions, without specific limitations. After triggering the first asynchronous event of a short-term frequent fault event or stability fault event in the system, if it is determined that image data can be obtained normally within a preset time, the normal working state of the image input / output system will be maintained, and the system will not be restarted. This ensures that the first target function in which the image input / output system participates can work normally without being degraded. After the first target function exits, restarting the image input / output system will no longer affect the first target function. At this point, restarting the image input / output system can be controlled to ensure that the faults corresponding to the previously triggered asynchronous events (including short-term frequent fault events or stability fault events) will not recur, ensuring the subsequent safe operation of the image input / output system and further improving vehicle driving safety.

[0074] In an optional example, after fault handling based on the first asynchronous event in step 207, the method further includes:

[0075] Step 208: In response to the failure of the fault handling, the target image link where the current fault is located is determined based on the fault information corresponding to the first asynchronous event.

[0076] If the fault handling result is failure, it means that the target image link where the current fault is located has experienced an unrecoverable fault. The target image link where the current fault is located can be determined based on the image link where the fault occurred, which is included in the fault information corresponding to the first asynchronous event.

[0077] Step 209: In response to the target image link having a pre-configured alternative image link and the alternative image link being in a non-faulty state, the environmental information of the viewpoint corresponding to the target image link is determined based on the image data obtained from the alternative image link and the alternative algorithm and / or alternative model corresponding to the pre-configured alternative image link.

[0078] This system allows for the pre-configuration of alternative image links for each image link. For instance, the image link corresponding to the front-view camera can be replaced by the image links corresponding to the left and right front-view cameras, respectively. This means that the image data covering the front-view range is determined by combining the left and right front-view images, allowing the image data to continue participating in functions requiring front-view image data in autonomous driving, ensuring that these functions can continue operating without degradation. Specific alternative image links can be set according to actual needs. The status of the alternative image links (including both non-fault and fault states) can be determined based on the real-time maintenance of the fault status of each image link. For example, the status of each image link can be maintained in real-time based on the aforementioned fault information and fault monitoring information, and can be set according to actual needs. The alternative algorithm and / or alternative model corresponding to the alternative image link refers to the perception algorithm or model for environmental perception based on the image data obtained from the alternative image link. This can be set according to actual needs; the perception model may include pre-trained object detection models, semantic segmentation models, etc., without specific limitations. Based on the corresponding alternative algorithms and / or alternative models, environmental information corresponding to the viewpoint of the target image link is extracted from the image data obtained from the alternative link image. Thus, based on the alternative image link and the corresponding algorithm model, the function of the target image link is completed.

[0079] When any target image link fails to handle a fault in the upper-layer application, this disclosure allows for the substitution of the function of that target image link by replacing the alternative image link. This ensures that the autonomous driving function in which the target image link participates can continue to operate without being downgraded, thereby improving the accuracy of the downgrade operation and further enhancing the user experience.

[0080] In an optional example, since the target image link does indeed fail and cannot be recovered when the alternative image link is running, and the effect of the alternative solution may differ from that of the target image link, fault alarm information can be controlled to be output when using the alternative image link to alert the user to the current system problem.

[0081] In one optional example, the autonomous driving function can be controlled at different levels based on the substitution effect level of the alternative image link. For example, the user can partially take over driving while retaining some autonomous driving functions. The specific settings can be configured according to actual needs.

[0082] In one optional example, the methods disclosed herein also include:

[0083] Step 210: In response to the target image link not having a pre-configured alternative image link or the pre-configured alternative image link being in a fault state, control the second target function involved by the target image link to be downgraded, and output downgrade prompt information.

[0084] The second target function is the function in which the target image link participates in the autonomous driving function, such as the autonomous driving behavior decision function. If the target image link does not have a pre-configured alternative image link or the pre-configured alternative image link is in a fault state, it means that the function replacement has failed. At this time, in order to ensure the safety of vehicle driving, it is necessary to control the second target function in which the target image link participates to be downgraded and output downgrade prompt information to promptly prompt the user to take over driving and avoid traffic accidents.

[0085] This disclosure only downgrades the autonomous driving function after ruling out all situations that would ensure its continued operation, thus avoiding accidental triggering of the downgrade operation and effectively improving the user experience.

[0086] In an optional example, after step 203, in response to the target level being a first preset level, performing fault recovery processing corresponding to the target type on the current fault, the method of this disclosure further includes:

[0087] Step 401: In response to the failure of the fault recovery process, a second asynchronous event is generated.

[0088] The second asynchronous event can be a fault recovery failure event. For a current fault of the first preset level, if recovery is not possible, it indicates that the fault recovery process has failed. Similar to the first asynchronous event, the second asynchronous event is used to trigger fault handling in the upper-layer application; further details will not be elaborated here.

[0089] Step 402: Perform fault handling based on the second asynchronous event.

[0090] Among them, the current fault cannot be self-recovered, but the fault can be handled through the upper-level application control, such as controlling the restart of the image input and output system. The specific settings can be configured according to actual needs.

[0091] This disclosure addresses situations where faults of the first preset level cannot be self-recovered. It also allows for real-time and rapid reporting to upper-layer applications via an asynchronous event mechanism, enabling the upper-layer applications to quickly handle the fault and resolve the problem promptly.

[0092] In an optional example, step 402, based on a second asynchronous event, performs fault handling, including: in response to a fault recovery handling failure event of the second asynchronous event, controlling the restart of the image link of the image input / output system.

[0093] For faults that cannot be self-recovered, the upper-layer application can resolve the current fault by controlling the restart of the image link of the image input / output system, so that the image link can quickly restore normal function and ensure the normal operation of the autonomous driving function.

[0094] In an optional example, an asynchronous fault handling application (or upper-layer asynchronous fault handling application or upper-layer asynchronous fault handling module) can be set up in the upper layer to respond to asynchronous events (including each first asynchronous event and second asynchronous event, and may also include other asynchronous events set according to actual needs) and quickly perform corresponding fault handling in the upper layer.

[0095] This disclosure classifies faults by level and type, enabling the collection and monitoring of faults at the underlying system software level. It blocks the reporting of most faults that can recover quickly on their own, and based on the collected fault monitoring information, it uses an asynchronous event mechanism to asynchronously report non-recoverable faults. This allows for real-time and rapid processing of asynchronous event faults, and in the event of fault handling failure or fault self-recovery failure, it implements functional substitution by replacing the image link to ensure that the autonomous driving function can continue to operate, effectively reducing the degradation rate of the autonomous driving function and improving the user experience.

[0096] The various embodiments and optional examples disclosed herein can be implemented individually or in any combination without conflict, and can be set according to actual needs.

[0097] Any of the image input / output system fault handling methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image input / output system fault handling methods provided in this disclosure can be executed by a processor, such as by a processor calling corresponding instructions stored in memory to execute any of the image input / output system fault handling methods mentioned in this disclosure. Further details will not be elaborated below.

[0098] Exemplary device

[0099] Figure 5 This is a schematic diagram of a fault handling apparatus for an image input / output system provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement corresponding method embodiments of this disclosure, such as... Figure 5 The apparatus shown includes: a first acquisition module 501, a first determination module 502, and a first processing module 503.

[0100] The first acquisition module 501 is used to acquire fault information of the image input / output system; the first determination module 502 is used to determine the target type and target level of the current fault based on the fault information; the first processing module 503 is used to perform fault recovery processing on the current fault corresponding to the target type in response to the target level being a first preset level.

[0101] Figure 6This is a schematic diagram of the structure of an image input / output system fault processing apparatus provided in another exemplary embodiment of the present disclosure.

[0102] In one optional example, the apparatus of this disclosure further includes:

[0103] The fault monitoring module 504 is used to acquire fault monitoring information of the image input / output system; the second determination module 505 is used to determine whether the asynchronous event triggering condition is met based on the fault monitoring information; the first generation module 506 is used to generate a first asynchronous event corresponding to the fault monitoring information in response to the fault monitoring information meeting the asynchronous event triggering condition, and report it to the first asynchronous fault handling module; the first asynchronous fault handling module 507 is used to perform fault handling based on the first asynchronous event.

[0104] Figure 7 This is a schematic diagram of the structure of the first asynchronous fault handling module 507 provided in an exemplary embodiment of this disclosure.

[0105] In one optional example, the first asynchronous fault handling module 507 includes:

[0106] The first processing unit 5071 is configured to respond to the first asynchronous event being a single fatal fault event and determining that the fault type corresponding to the first asynchronous event is a physical fault, and control the output of fault prompt information; the second processing unit 5072 is configured to respond to the first asynchronous event being a single fatal fault event and determining that the fault type corresponding to the first asynchronous event is a software fault, and control the restart of the image link of the image input / output system; the third processing unit 5073 is configured to respond to the first asynchronous event being a short-term frequent fault event or a stability fault event, and determine whether image data can be obtained within a preset time; the fourth processing unit 5074 is configured to respond to the image data being obtained within the preset time and maintain the normal working state of the image input / output system; the fifth processing unit 5075 is configured to respond to the image data not being obtained within the preset time and control the restart of the image link of the image input / output system.

[0107] In an optional example, the apparatus of this disclosure further includes a sixth processing unit 5076, configured to control the restart of the image input / output system in response to the exit of a first target function in which the image input / output system participates.

[0108] In one optional example, the apparatus of this disclosure further includes:

[0109] The second processing module 508 is used to determine the target image link where the current fault is located based on the fault information corresponding to the first asynchronous event in response to the failure of the fault handling process; the third processing module 509 is used to determine the environmental information of the viewpoint corresponding to the target image link based on the image data obtained from the alternative image link and the alternative algorithm and / or alternative model corresponding to the pre-configured alternative image link in response to the target image link having a pre-configured alternative image link and the alternative image link being in a non-fault state.

[0110] In an optional example, the apparatus of this disclosure further includes: a fourth processing module 601, configured to control the second target function involved by the target image link to be downgraded in response to the target image link not having a pre-configured alternative image link or the pre-configured alternative image link being in a fault state, and to output downgrade prompt information.

[0111] In one optional example, the apparatus of this disclosure further includes:

[0112] The second generation module 602 is used to generate a second asynchronous event in response to the failure of the fault recovery process; the second asynchronous fault handling module 603 is used to perform fault handling based on the second asynchronous event.

[0113] The second asynchronous fault handling module 603 and the aforementioned first asynchronous fault handling module 507 can be the same module or two independent modules, depending on actual needs.

[0114] In an optional example, the second asynchronous fault handling module 603 is specifically used to control the restart of the image link of the image input / output system in response to the second asynchronous event being a fault recovery processing failure event.

[0115] In one optional example, Figure 8This is a schematic diagram of the framework of an autonomous driving behavior decision-making function system provided by an exemplary embodiment of the present disclosure. In this example, the autonomous driving behavior decision-making function system includes an upper-layer application, a hardware abstraction layer, a hardware driver layer, and hardware components. The image input / output system fault handling device of the present disclosure (part 1, which may include the aforementioned first acquisition module, first determination module, first processing module, fault monitoring module, second determination module, first generation module, second generation module, etc., located in the hardware abstraction layer; and part 2, which may include the aforementioned first asynchronous fault handling module and second asynchronous fault handling module, located in the upper-layer application) is set up in the hardware abstraction layer and the upper-layer application to complete the image input / output system fault handling method of the present disclosure. The upper-layer application, in addition to the device part 2 of the present disclosure, may also include various upper-layer applications of autonomous driving functions, such as perception applications based on image data for environmental perception, which will not be elaborated further. The hardware may include an image sensor, an ISP unit for signal processing of the raw data acquired by the image sensor, various hardware processing units for image processing of the image data, etc. The hardware driver layer includes software programs for driving various hardware components. The hardware abstraction layer is a software library abstracted from the hardware driver layer, containing commonalities among the hardware drivers. Libcam represents the software library related to image sensors (cameras), and LIBVIO represents the software library related to the VIO system. Each module in device part 1 monitors and records hardware and software-related faults of the image sensor and VIO system. For faults that can be quickly and autonomously recovered at the lower level, device part 1 controls or indirectly controls the autonomous and rapid recovery of the lower level. For faults that cannot be quickly and autonomously recovered at the lower level, asynchronous events (including the first and second asynchronous events mentioned above) are generated in real time and reported to device part 2 of the upper-layer application in real time through an asynchronous mechanism. Each module in device part 2 is responsible for responding to each asynchronous event and performing corresponding asynchronous fault handling in real time at the upper layer, such as controlling the restart of the VIO system, controlling the replacement of the faulty target image link with an alternative image link, etc., to ensure that the normal operation of the autonomous driving function is not affected. When it is determined that fault handling has failed and functional replacement is not possible, or when other unsolvable faults occur, a diagnostic report is made, the corresponding autonomous driving function is downgraded, and the user is notified. Specific handling for various situations is described above and will not be repeated here.

[0116] Exemplary electronic devices

[0117] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0118] A processor is configured to execute a computer program stored in the memory, and when the computer program is executed, to implement the image input / output system fault handling method described in any of the above embodiments of the present disclosure.

[0119] Figure 9 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0120] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0121] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0122] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0123] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0124] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0125] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0126] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0127] Exemplary computer program products and computer-readable storage media

[0128] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0129] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0130] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0131] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0132] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0134] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0135] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0136] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0137] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0138] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for handling faults in an image input / output system, comprising: The fault information of the image input / output system is obtained. The image input / output system is used to process the raw data collected by the image sensor to obtain image data for environmental perception. Based on the fault information, determine the target type and target level to which the current fault belongs; In response to the target level being a first preset level, the current fault undergoes fault self-recovery processing corresponding to the target type; wherein, the first preset level is a level where the fault is self-recoverable; The method further includes: Obtain fault monitoring information of the image input / output system; Based on the fault monitoring information, determine whether the asynchronous event triggering conditions are met; In response to the fault monitoring information satisfying the asynchronous event triggering condition, a first asynchronous event corresponding to the fault monitoring information is generated. Based on the first asynchronous event, perform fault handling; If the processing result of the fault handling is failure, the target image link where the current fault is located is determined based on the fault information corresponding to the first asynchronous event. In response to the target image link having a pre-configured alternative image link and the alternative image link being in a non-faulty state, the environmental information of the viewpoint corresponding to the target image link is determined based on the image data acquired by the alternative image link and the pre-configured alternative algorithm and / or alternative model corresponding to the alternative image link.

2. The method according to claim 1, wherein, The fault handling based on the first asynchronous event includes: In response to the first asynchronous event being a single fatal fault event, and determining that the fault type corresponding to the first asynchronous event is a physical fault, the control outputs a fault prompt message; In response to the first asynchronous event being a single fatal failure event, and determining that the failure type corresponding to the first asynchronous event is a software failure, the image link of the image input / output system is restarted. In response to the first asynchronous event being a short-term frequent fault event or a stability fault event, determine whether image data can be obtained within a preset time. The image data can be obtained within the preset time, and the normal working state of the image input / output system is maintained. If the image data cannot be obtained within the preset time, the image link of the image input / output system is restarted.

3. The method according to claim 2, wherein, After the image data can be obtained within the preset time and the normal operation of the image input / output system is maintained, the method further includes: In response to the exit of the first target function in which the image input / output system participates, the image input / output system is restarted.

4. The method according to claim 1, further comprising: In response to the absence of a pre-configured alternative image link for the target image link or the pre-configured alternative image link being in a fault state, the second target function in which the target image link participates is degraded, and a degradation prompt message is output.

5. The method according to claim 1, wherein, After responding to the target level being a first preset level and performing fault recovery processing corresponding to the target type on the current fault, the method further includes: If the fault recovery process fails, a second asynchronous event is generated. Based on the second asynchronous event, perform fault handling.

6. The method according to claim 5, wherein, The fault handling based on the second asynchronous event includes: In response to the second asynchronous event being a fault recovery failure event, the image link of the image input / output system is restarted.

7. A fault handling device for an image input / output system, comprising: The first acquisition module is used to acquire fault information of the image input / output system, wherein the image input / output system is used to process the raw data collected by the image sensor to obtain image data for environmental perception. The first determining module is used to determine the target type and target level of the current fault based on the fault information; The first processing module is configured to, in response to the target level being a first preset level, perform fault self-recovery processing on the current fault corresponding to the target type; wherein, the first preset level is a level at which the fault can recover on its own; The device further includes: The fault monitoring module is used to acquire fault monitoring information of the image input / output system; The second determining module is used to determine whether the asynchronous event triggering conditions are met based on the fault monitoring information. The first generation module is configured to generate a first asynchronous event corresponding to the fault monitoring information in response to the fault monitoring information satisfying the asynchronous event triggering condition. The first asynchronous fault handling module is used to handle faults based on the first asynchronous event; The second processing module is used to determine the target image link where the current fault is located based on the fault information corresponding to the first asynchronous event in response to the failure of the fault handling process. The third processing module is used to determine the environmental information of the viewpoint corresponding to the target image link in response to the fact that the target image link has a pre-configured alternative image link and the alternative image link is in a non-faulty state, based on the image data obtained by the alternative image link and the pre-configured alternative algorithm and / or alternative model corresponding to the alternative image link.

8. A computer-readable storage medium storing a computer program for executing the image input / output system fault handling method according to any one of claims 1-6.

9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image input / output system fault handling method according to any one of claims 1-6.

Images