Camera fault detection method and device, equipment and storage medium
By integrating fault detection components in terminal equipment, fast and accurate detection of camera failures is achieved, and the problem of untimely detection of camera failures in the prior art is solved, ensuring the safe operation of the autonomous driving system.
Patent Information
- Application Number
- CN202311542467.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-17
- Publication Date
- 2025-05-20
AI Technical Summary
The lack of effective camera fault detection methods in the prior art leads to the inability of automatic driving systems to detect and handle camera failures in time, which may lead to safety hazards.
The fault detection of the camera is achieved by integrating the fault detection component, including the camera drive component and the register component, in the terminal device. Specific methods include calling the program class in the camera driver component to detect deserializer hardware failure, camera outflow failure and frame failure, and accessing the register address through the register component to read the register status to detect failures of the camera module, deserializer communication, serializer, power supply chip and memory chip.
It realizes fast and accurate detection of camera failures, can report fault information in a timely manner, and ensures the safe operation of the autonomous driving system and the safety and reliability of driving.
Smart Images
Figure CN120021247A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of terminals, and in particular, to a camera fault detection method, apparatus, device, and storage medium. Background Art
[0002] An autonomous driving camera is a core component of an autonomous driving system, which realizes the autonomous driving function by collecting image or video data around the vehicle. If the camera fails or is damaged, it will affect the normal operation of the autonomous driving system, and in severe cases, it may even lead to accidents. Currently, effective detection methods for camera faults are not yet mature. Summary of the Invention
[0003] To overcome the problems in the related art, the present disclosure provides a camera fault detection method, apparatus, device, and storage medium, which can effectively and accurately detect the fault information of the camera.
[0004] According to a first aspect of an embodiment of the present disclosure, a camera fault detection method is provided. The camera includes a fault detection component, and the method includes:
[0005] Performing fault detection on the camera based on the fault detection component to obtain the fault information of the camera;
[0006] Wherein, when the fault detection component includes a camera driver component, the fault information includes at least one of a deserializer hardware fault, a camera data stream output fault, and a camera frame fault; when the fault detection component includes a register component, the fault information includes at least one of an imaging module fault, a deserializer communication fault, a serializer fault, a power supply chip fault, and a storage chip fault.
[0007] In some embodiments, if the fault detection component includes the camera driver component, then performing fault detection on the camera based on the fault detection component to obtain the fault information of the camera includes at least one of the following:
[0008] Invoking a first program class in the camera driver component to perform abnormal listening on the deserializer in the camera to capture the deserializer hardware fault of the camera;
[0009] Invoking a second program class in the camera driver component to detect whether the data stream of the camera is received within a preset first time period to determine whether the camera data stream output fault occurs;
[0010] Invoking a third program class in the camera driver component to detect whether the camera frame fault occurs;
[0011] When the data stream of the camera is not received within the preset first duration, it is determined that a camera out - of - stream fault occurs. The camera frame fault includes at least one of the following faults: frame loss, non - consecutive frames, frame capture processing ICP timeout, and frame format error.
[0012] In some embodiments, if the fault detection component includes the register component, then the fault detection of the camera based on the fault detection component to obtain the fault information of the camera includes:
[0013] Based on the handle corresponding to the register component, access the address of the register component and read the register status of the register component. The register status is at least used to reflect whether the camera has a fault;
[0014] Based on the register status of the register component, obtain the fault information of the camera.
[0015] In some embodiments, the obtaining the fault information of the camera based on the register status of the register component includes:
[0016] When the register status of the register component is not the preset status, it is determined that the camera has a fault and the fault information of the camera is obtained.
[0017] In some embodiments, the method further includes:
[0018] When the consecutive occurrence times of the fault information are greater than or equal to the preset times, report the fault information.
[0019] In some embodiments, when there are N cameras in N imaging ranges connected to N deserializers, and each deserializer is connected to M device control chips, there are N×M deserializer communication faults. Each deserializer communication fault is used to indicate that the data stream of the camera in the corresponding imaging range has a communication fault. Both N and M are positive integers.
[0020] In some embodiments, the power supply chip fault includes any one of the following faults: open circuit, short circuit, over - voltage, and under - voltage.
[0021] In some embodiments, the method further includes:
[0022] After the camera has a fault, perform fault recovery processing on the camera based on the set state of the finite state machine. The set state is used to indicate the current state of the camera.
[0023] In some embodiments, the performing fault recovery processing on the camera based on the set state of the finite state machine includes:
[0024] After the camera fails, set the current state of the camera to an unavailable state and power off the camera.
[0025] After the power-off process, power on the camera.
[0026] After the camera is successfully powered on, set the current state of the camera to an available state to indicate that the fault recovery process for the camera is successful.
[0027] In some embodiments, before the power-on process for the camera, the method further includes:
[0028] When powering off the camera, set the current state of the camera to a transition state, where the transition state is used to indicate that the camera will be powered on after waiting for a preset second duration; and / or,
[0029] After the power-off process for the camera, set the current state of the camera to a power-on state, where the power-on state is used to indicate that the camera will be powered on.
[0030] According to a second aspect of the embodiments of the present disclosure, there is provided a camera fault detection device. The camera includes a fault detection component, and the device includes:
[0031] A processing module configured to perform fault detection on the camera based on the fault detection component to obtain fault information of the camera.
[0032] Wherein, when the fault detection component includes a camera driving component, the fault information includes at least one of a deserializer hardware fault, a camera output stream fault, and a camera frame fault; when the fault detection component includes a register component, the fault information includes at least one of an imaging module fault, a deserializer communication fault, a serializer fault, a power supply chip fault, and a storage chip fault.
[0033] For the content not introduced or described in the embodiments of the present disclosure, reference may be made to the relevant introduction in the foregoing method embodiments, and the embodiments of the present disclosure do not make any limitations.
[0034] According to a third aspect of the embodiments of the present disclosure, there is provided a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the executable instructions to implement the steps of the above-mentioned camera fault detection method.
[0035] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the camera fault detection method provided in the first aspect of the present disclosure are implemented.
[0036] According to a fifth aspect of the embodiments of the present disclosure, a chip is provided, including: a processor and an interface; the processor is configured to read instructions to execute the steps of the above-mentioned camera fault detection method.
[0037] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: The terminal device performs fault detection on the camera based on the fault detection component to obtain the fault information of the camera; the above-mentioned fault detection component may include a camera driver component and / or a register component, wherein when the fault detection component includes the camera driver component, the fault information includes at least one of a deserializer hardware fault, a camera output fault, and a camera frame fault; when the fault detection component includes the register component, the fault information includes at least one of an imaging module fault, a deserializer communication fault, a serializer fault, a power supply chip fault, and a storage chip fault. It can be seen that the terminal device can implement the fault detection of the camera based on the camera driver component and / or the register component, so as to obtain the fault information of the camera. In this way, the camera fault can be detected more comprehensively, more timely and effectively, and then repaired to ensure the safe operation of the subsequent autonomous driving system and guarantee the safety and reliability of driving.
[0038] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.
[0040] Figure 1 is a flowchart showing a camera fault detection method according to an exemplary embodiment.
[0041] Figure 2 is a schematic diagram showing fault detection based on a camera driver component according to an exemplary embodiment.
[0042] Figure 3 is a schematic diagram showing fault detection based on a register component according to an exemplary embodiment.
[0043] Figure 4 is a schematic diagram showing a scenario of camera fault recovery according to an exemplary embodiment.
[0044] Figure 5 is a schematic diagram showing the structure of a camera fault detection device according to an exemplary embodiment.
[0045] Figure 6 is a schematic diagram showing the structure of a terminal device according to an exemplary embodiment.
[0046] Figure 7 It is a schematic structural diagram of a chip shown according to an exemplary embodiment. Detailed implementation manners
[0047] Here, the exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0048] It should be noted that all actions of obtaining signals, information, or data in the present disclosure are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining authorization from the owner of the corresponding device.
[0049] An autonomous driving camera needs to have a fault detection function (which can also be called a fault diagnosis function or a diagnosis function) to ensure its normal operation and timely detection of faults. It has been found in practice that the current camera functions can use underlying driver components (such as SIPL components, etc.). Since the underlying driver SIPL component is a general software component, it can only support some general camera fault detection functions; for example, some software problems: no video stream in the camera, frame loss in the camera, or no signal in the camera, etc. However, for some hardware problems, they cannot be achieved through the SIPL component. Based on this, the present disclosure proposes a camera fault detection method, device, equipment, and storage medium. For scenarios that cannot be covered by the SIPL component, the idea of a register component can be used to implement the functions of other hardware fault diagnosis / detection, so that the advantages of the SIPL component's fault detection, such as stable and reliable, low functional load, and timely problem response, can be retained, and the fault detection of other hardware can also be achieved.
[0050] Please refer to Figure 1 It is a schematic flowchart of a camera fault detection method shown according to an exemplary embodiment. As Figure 1 shown, the method can be applied to a terminal device, and the terminal device is installed or provided with a fault detection component; or the camera of the terminal device includes / has a fault detection component. As Figure 1 shown, the method may include the following implementation steps:
[0051] S101. Perform a fault detection on the camera based on the fault detection component to obtain the fault information of the camera. Among them, when the fault detection component includes the camera driver component, the fault information includes at least one of the deserializer hardware fault, the camera output fault, and the camera frame fault. When the fault detection component includes the register component, the fault information includes at least one of the imaging module fault, the deserializer communication fault, the serializer fault, the power supply chip fault, and the storage chip fault.
[0052] The above-mentioned fault detection component in the present disclosure refers to a component installed / set in a terminal device or its camera for fault detection or diagnosis, which may include a camera driver component and / or a register component. That is to say, the above-mentioned fault detection component may include a camera driver component, may include a register component, or may include both a camera driver component and a register component. Among them, the above-mentioned camera driver component may refer to a camera low-level driver component, such as a SIPL component or other software components with camera fault detection capabilities. The above-mentioned register component may also be simply referred to as a register. The present disclosure does not limit the respective numbers of the above-mentioned camera driver component and the above-mentioned register component, which can be determined according to actual situations. Generally, the number of the above-mentioned camera driver components is one, and the number of the above-mentioned register components may be one or more. The present disclosure does not make excessive limitations and details on this.
[0053] The present disclosure can use the above-mentioned fault detection component to perform a fault detection on the camera, so as to determine whether the camera has a fault. When a fault occurs, the fault information of the corresponding camera can be obtained. Among them, the present disclosure does not limit the specific implementation manner of the above-mentioned fault detection, and the following is an exemplary introduction to its content.
[0054] In one embodiment, when the above-mentioned fault detection component includes the above-mentioned camera driver component, the specific implementation manner of performing a fault detection on the camera based on the camera driver component may include, but is not limited to, any one or a combination of the following several ways:
[0055] In one implementation, the present disclosure can call a first program class in the camera driver component to perform exception monitoring on the deserializer in the camera to capture hardware failures of the camera's deserializer. The above-mentioned first program class can refer to a program code class set in the camera for deserializer hardware failure / exception monitoring. For example, it can be a device block notification class such as DeviceBlockNotification. In a specific implementation, when the present disclosure calls a first program class such as the above-mentioned DeviceBlockNotification, a thread will be started to continuously monitor the exceptions of the camera in the background. The internal implementation mechanism is that various hardware in the device will generate abnormal PIN corner voltages during exceptions, actively triggering the first program class such as the above-mentioned DeviceBlockNotification to receive the exception messages of the corresponding hardware, and at the same time, it can also detect which type of hardware failure has occurred. For example, it can detect which type of deserializer has failed, etc. The present disclosure does not make too many limitations and details on this.
[0056] In another implementation, the present disclosure can call a second program class in the camera driver component to detect whether it receives the data stream of the camera within a preset first time period, so as to determine whether there is a camera data output failure (that is, to determine whether the camera has a data output failure). Specifically, when calling the above-mentioned second program class and not receiving the data stream of the camera within the preset first time period, it can be determined that there is a camera data output failure; on the contrary, when receiving the data stream of the camera within the preset first time period, it can be determined that there is no camera data output failure. The above-mentioned second program class can refer to a program code class set in the camera for detecting camera data output failures. For example, it can be a frame queue processing class such as FrameQueueHandler. In a specific implementation, when the present disclosure calls a second program class such as the above-mentioned FrameQueueHandler, a thread will be started to wait for the arrival of the camera data stream (which can also be simply referred to as camera data) within the preset first time period. The above-mentioned preset first time period is a waiting time threshold for camera data output (or camera data) custom-set by the terminal device or user according to the actual situation. For example, it can be an empirical value set according to user experience, or a statistical value calculated based on a series of experimental data, etc. It can be understood that usually the normal image output time of the camera is 33 milliseconds. Therefore, the above-mentioned preset first time period usually needs to be greater than 33 milliseconds. Taking the above-mentioned preset first time period as 100 milliseconds as an example, if the present disclosure does not receive the data stream of the camera within 100 milliseconds, it can be determined that there is a camera data output failure, and further, the failure information can be reported to trigger the camera function degradation. For example, any function execution based on the data stream of the camera can be blocked subsequently, such as blocking the subsequent execution of autonomous driving, etc.
[0057] In yet another embodiment, the present disclosure may call a third program class in the camera driver component to detect whether a camera frame failure occurs. The camera frame failure may include, but is not limited to, any one or a combination of the following failures: frame loss, non - consecutive frames, frame capture processing (Image Capture Pipeline, ICP) timeout, frame format error, or other failures related to camera frames or data streams, etc. The above - mentioned third program class may refer to a program code class set in the camera for detecting camera frame failures. For example, it may be a PipelineNotifier class such as a pipeline notification class. In a specific implementation, when the present disclosure calls a third program class such as PipelineNotifier, it also starts a background thread to continuously query the event queue of the camera pipeline to determine whether a corresponding camera frame failure occurs. Among them, the above - mentioned third program class (such as PipelineNotifier) supports the detection of many event failures, such as frame loss, frame discontinuity (i.e., non - consecutive frames appear), ICP timeout, incorrect data stream (i.e., image frame) of the camera, etc. Since the problems or impacts brought by frame loss failures are relatively large, the present disclosure can mainly monitor frame loss failures among the above - mentioned camera frame failures. The above - mentioned ICP can also be called an image capture pipeline, and ICP timeout may refer to the timeout of the acquisition and processing time of the camera data stream. Usually in the ICP, the optical sensor of the camera can first read the raw image data and then perform a series of visualization processes on these raw image data to generate the finally visualized image data, etc.
[0058] For example, please refer to Figure 2 is a schematic diagram of a fault detection based on a camera driver component shown according to an exemplary embodiment. Combining Figure 2 , the present disclosure takes the first program class as the DeviceBlockNotification class, the second program class as the FrameQueueHandler class, and the third program class as the PipelineNotifier class as examples to illustrate the specific implementation process of the camera driver component for fault detection. Among them, the illustration is only an example and does not constitute a limitation.
[0059] In yet another embodiment, when the above-mentioned fault detection component includes the above-mentioned register component, the specific implementation manner of performing fault detection on the camera based on the register component may include, but is not limited to, for example, accessing the address of the register component based on the handle corresponding to the register component, and then reading the register status of the register component. The register status is at least used to reflect whether the camera has a fault, and further optionally can also be used to reflect the specific fault that occurs in the camera, that is, to reflect the specific fault information of the camera, etc. Among them, the present disclosure does not limit the number of the above-mentioned register components, which can be determined according to actual situations. For example, it may include, but is not limited to, any one or a combination of the following register components: serializer (Ser), imaging module (Sensor), storage chip (Eeprom), deserialzier (DeSer), and power supply chip (also referred to as a power chip, Campwr). In specific implementation, each register component will expose a corresponding handle to the application layer of the terminal device in the driver, and the content of the register component can be read and updated by obtaining / accessing the handle of the corresponding register component. That is, the address of the register component is accessed through the handle corresponding to the register component, and then the register status and other contents stored in the register component are read.
[0060] After obtaining the register status of the above-mentioned register component, the present disclosure can obtain the fault information of the corresponding camera based on the register status of the above-mentioned register component. In specific implementation, the present disclosure can determine whether the register status of the above-mentioned register component is a preset status to determine whether the camera has a fault, and then obtain the fault information of the camera, etc. The present disclosure does not limit the specific manifestation form of the register status of the above-mentioned register component. For example, it can be represented in the form of numbers, letters, characters, strings, or a combination thereof, etc. When the register status of the above-mentioned register component is the preset status, it can be determined that the camera has no fault, and thus the process can be ended. On the contrary, when the register status of the above-mentioned register component is not the preset status, it can be determined that the camera has a fault, and the specific fault information of the camera can be obtained based on the indication of the register status. For example, please refer to Figure 3 is a schematic diagram of performing fault detection based on a register component shown according to an exemplary embodiment. As Figure 3 , for example, in the initialization stage, a background thread such as "analyse.DoAnalyse" or "ADOS / DIA" can be started to read the register addresses of all register components within a preset second duration (such as 100 milliseconds), and then determine / determine whether the register component has the fault information of the corresponding camera, etc. For example, in the figure, the background thread can access the register addresses of 5 register components, namely serializer (Ser), imaging module (Sensor), storage chip (Eeprom), deserialzier (DeSer), and power supply chip (Campwr). The present disclosure does not make too many limitations and details on this.
[0061] In practical applications, when the above register component is a serializer (Ser) register component, the present disclosure can detect the serializer fault in the camera by using the above serializer (Ser) register component according to the above fault detection principle. Similarly, when the above register component is an imaging module (Sensor) register component, the present disclosure can detect the imaging module fault in the camera by using the above imaging module (Sensor) register component according to the above fault detection principle. When the above register component is a storage chip (Eeprom) register component, the present disclosure can detect the storage chip fault in the camera by using the above storage chip (Eeprom) register component according to the above fault detection principle. When the above register component is a deserialization (DeSer) register component, the present disclosure can detect the deserialization communication fault in the camera by using the above deserialization (DeSer) register component according to the above fault detection principle. When the above register component is a power supply chip (Campwr) register component, the present disclosure can detect the power supply chip fault in the camera by using the above power supply chip (Campwr) register component according to the above fault detection principle. The specific implementation of the fault detection can be referred to the relevant introduction in the foregoing embodiments, which will not be elaborated here. Exemplarily, taking the fault detection or diagnosis of the deserialization communication fault and the power supply chip fault as examples, the specific embodiments involved are introduced below.
[0062] In serializer communication failure detection, such as serializer Inter-Integrated Circuit (I2C) communication failure detection, cameras within different imaging ranges can be connected to different serializers. For example, N cameras within different imaging ranges can be connected to N serializers, and all cameras within one imaging range are correspondingly connected to one serializer. Moreover, each serializer supports connection to all or some of the device control chips in the terminal device. For example, in this disclosure, each serializer can be connected to M device control chips. In this case, the number of serializer communication failures can be N×M, that is, there can be N×M reports of serializer communication failure (information). Each serializer communication failure can be used to indicate that a communication failure has occurred in the data stream of the camera within the corresponding shooting range, that is, the corresponding device control chip cannot obtain the data stream of the camera within the corresponding shooting range, etc. Both N and M are positive integers custom-set according to actual situations. For example, taking N equal to 3 and M equal to 2 as an example, there are 3 serializers deployed on the camera platform, namely Serializer 1, Serializer 2, and Serializer 3. Among them, Serializer 1 is connected to the cameras within the front and rear view shooting ranges, Serializer 2 is connected to the cameras within 4 side view ranges, and Serializer 3 is connected to the cameras within the panoramic shooting range. Each serializer is connected to 2 device control chips in the terminal device, which can be SOC1 and SOC2 respectively; in this case, there can be 6 reports of I2C communication failures of the serializer. In the specific detection of serializer I2C communication failure, the address of each serializer (DeSer) register component (such as 0xD) can be accessed, and then the register status stored at this register address can be read. When the register status is not the preset status (such as 0x01), it can be determined that the above-mentioned serializer I2C communication failure has occurred, etc.
[0063] In power supply chip failure detection, the specific failures (information) it supports can include, but are not limited to, any one or a combination of the following: open circuit, short circuit, overvoltage, undervoltage, or other failures that may occur in custom power supply chips or power supply circuits, etc. In the specific detection of power supply chip failure, it can be determined / judged whether the corresponding power supply chip failure has occurred by accessing the Campwr register component. This disclosure does not make excessive limitations and details here, and it can refer to the relevant introduction in the foregoing embodiments correspondingly.
[0064] In some alternative embodiments, to ensure the reliability of fault detection / diagnosis, the present disclosure may adopt a detection debounce strategy to reduce false alarms caused by occasional events. In specific implementation, the present disclosure may report the above-mentioned fault information only when the consecutive occurrence times of the fault information of the above-mentioned camera are greater than or equal to a preset number of times. Conversely, when the consecutive occurrence times of the above-mentioned fault information are less than the preset number of times, it may be considered an accidental fault and no fault reporting is performed. The above-mentioned preset number of times may be a positive integer custom-set according to the actual situation. For example, the present disclosure may read the corresponding register components every 100 milliseconds regularly. If the same fault information is continuously found 3 times, a DTC fault information may be reported. Another example is that the present disclosure may detect whether the data stream of the camera is received (i.e., whether the camera outputs images) every 100 milliseconds regularly. If the data stream of the camera is not received (i.e., the camera does not output images), a no-signal event may be triggered. If the data stream of the camera is not received (i.e., the camera does not output images) for 3 consecutive times, a camera output stream fault may be reported, specifically, a no-signal fault may be reported, etc.
[0065] In still some alternative embodiments, after the camera fails (or is abnormal), the output frame rate of the camera will also be abnormal. That is to say, the present disclosure may determine whether the camera fails or is abnormal by counting the camera frame rate. Specifically, for example, when the camera frame rate is not within the preset frame rate range, it may be determined that the camera fails; conversely, it may be determined that the camera does not fail. The preset frame rate range is a normal frame rate range custom-set according to the actual situation, and the present disclosure does not make too many limitations and detailed descriptions on this. After the camera fails, the present disclosure may start a camera recovery mechanism to perform fault recovery processing on the camera. For example, please refer to Figure 4 is a schematic diagram of a camera fault recovery scenario shown according to an exemplary embodiment. As Figure 4 shown, when the current frame rate of camera 1 is 30 Hz, it can be determined that camera 1 does not fail; while when the current frame rate of camera 2 is 0 Hz, the autonomous driving system can determine that camera 2 fails, and then can start the internal camera recovery mechanism to perform fault / abnormality recovery processing on camera 2, etc. Among them, the above-mentioned camera recovery mechanism may be a fault recovery mechanism custom-set according to the actual situation. For example, the present disclosure may perform fault recovery processing on the camera by using / based on the set state of a finite state machine, and this set state can be used to indicate the current state of the camera.
[0066] The present disclosure does not limit the specific implementation manners of the above-mentioned fault recovery process. For example, in a possible implementation manner, after determining that the above-mentioned camera fails, the present disclosure may set the current state of the camera to the unavailable state (Disable) and perform a power-off process on the camera (which can also be referred to as a power-down process). After the power-off process, after a period of time, the camera can be powered on again. After the camera is successfully powered on, the current state of the camera can be set to the available state (which can also be referred to as the no-abnormality state, Null) to indicate that the above-mentioned fault recovery process of the camera is successful, the camera has no faults or abnormalities, and is currently available, etc. In some optional embodiments, when the present disclosure performs a power-off process on the camera, it can simultaneously set the current state of the above-mentioned camera to the transition state (Check), and this transition state is used to indicate that the camera will be powered on again after waiting for a preset second duration (i.e., waiting for a period of time). The preset second duration can be a time duration custom-set according to the actual situation. For example, it can be an empirical value set according to user experience, or a statistical value calculated based on a series of experimental data, etc. Further optionally, after the present disclosure performs a power-off process on the above-mentioned camera and before powering on the camera, the present disclosure can set the current state of the above-mentioned camera to the power-on state (Enable), and this power-on state is used to indicate that the camera is to be powered on, etc.
[0067] In still some other optional embodiments, after the present disclosure performs a power-on process on the camera, it can determine whether the camera is successfully powered on. After determining that the camera is successfully powered on, the current state of the camera can be set to the available state to indicate that the fault recovery process of the camera is currently completed, etc. Otherwise, when the camera is not successfully powered on, the process can be ended. Among them, the present disclosure does not limit the specific determination implementation manner of whether the above-mentioned camera is successfully powered on. For example, the present disclosure can determine by detecting whether the data stream of the camera can be received. Specifically, if the data stream of the camera can be received, it can be determined that the camera is successfully powered on, etc. Another example is that it can be determined by judging whether the camera frame rate is within the corresponding preset frame rate range. Specifically, if it is within the corresponding preset frame rate range, it can be determined that the camera is successfully powered on; otherwise, it can be determined that the camera is not successfully powered on or fails, etc.
[0068] By implementing the embodiments of the present disclosure, the present disclosure can quickly locate the fault information of the camera. For example, it can detect camera faults within milliseconds. Optionally, system alarms can also be performed, triggering function degradation to avoid accidents. In addition, the present disclosure can accurately locate the specific faults of the camera. For example, it can locate which camera module has a fault or what specific fault has occurred, such as module faults, communication faults, or power supply chip faults of the camera module, etc., which is convenient for subsequent maintenance. In specific implementation, the terminal device performs fault detection on the camera based on the fault detection component to obtain the fault information of the camera. The above-mentioned fault detection component includes a camera driver component and / or a register component. Among them, when the fault detection component includes the camera driver component, the fault information includes at least one of serializer hardware faults, camera out-flow faults, and camera frame faults. When the fault detection component includes the register component, the fault information includes at least one of camera module faults, serializer communication faults, serializer faults, power supply chip faults, and storage chip faults. It can be seen that the terminal device can implement fault detection of the camera based on the camera driver component and / or the register component, so as to obtain the fault information of the camera. In this way, camera faults can be detected more comprehensively, more timely and effectively, and then repaired to ensure the safe operation of the subsequent autonomous driving system and guarantee the safety and reliability of driving.
[0069] Based on the foregoing embodiments, please refer to Figure 5 is a schematic structural diagram of a camera fault detection device shown according to an exemplary embodiment. As Figure 5 shown, the device can be applied to a terminal device. The terminal device or the camera of the terminal device may include a fault detection component, and the fault detection component includes a camera driver component and / or a register component. As Figure 5 shown, the device may include a processing module 501. Among them:
[0070] The processing module 501 is configured to perform fault detection on the camera based on the fault detection component to obtain the fault information of the camera;
[0071] Among them, when the fault detection component includes the camera driver component, the fault information includes at least one of serializer hardware faults, camera out-flow faults, and camera frame faults. When the fault detection component includes the register component, the fault information includes at least one of camera module faults, serializer communication faults, serializer faults, power supply chip faults, and storage chip faults.
[0072] In some embodiments, if the fault detection component includes the camera driver component, the processing module 501 is configured to perform at least one of the following:
[0073] Call the first program class in the camera driver component to perform exception monitoring on the deserializer in the camera to capture hardware faults of the deserializer in the camera;
[0074] Call the second program class in the camera driver component to detect whether the data stream of the camera is received within a preset first duration to determine whether a camera out - flow fault occurs;
[0075] Call the third program class in the camera driver component to detect whether a camera frame fault occurs;
[0076] Among them, when the data stream of the camera is not received within the preset first duration, it is determined that a camera out - flow fault occurs, and the camera frame fault includes at least one of the following faults: frame loss, non - consecutive frames, frame capture processing ICP timeout, and frame format error.
[0077] In some embodiments, if the fault detection component includes the register component, the processing module 501 is configured to access the address of the register component based on the handle corresponding to the register component, and read the register status of the register component, where the register status is at least used to reflect whether a fault occurs in the camera; based on the register status of the register component, obtain the fault information of the camera.
[0078] In some embodiments, the processing module 501 is configured to determine that a fault occurs in the camera and obtain the fault information of the camera when the register status of the register component is not in the preset state.
[0079] In some embodiments, the processing module 501 is further configured to report the fault information when the consecutive occurrence times of the fault information are greater than or equal to the preset times.
[0080] In some embodiments, when N cameras within N camera ranges are connected to N deserializer, and each deserializer is connected to M device control chips, there are N×M deserializer communication faults, where each deserializer communication fault is used to indicate that a communication fault occurs in the data stream of the camera within the corresponding camera range, and both N and M are positive integers.
[0081] In some embodiments, the power supply chip fault includes any one of the following faults: open circuit, short circuit, over - voltage, and under - voltage.
[0082] In some embodiments, the processing module 501 is further configured to perform fault recovery processing on the camera based on the set state of the finite state machine after the camera fails, where the set state is used to indicate the current state of the camera.
[0083] In some embodiments, the processing module 501 is configured to:
[0084] After a failure occurs in the camera, set the current state of the camera to an unavailable state, and perform a power-off process on the camera;
[0085] After the power-off process, perform a power-on process on the camera;
[0086] After the camera is successfully powered on, set the current state of the camera to an available state to indicate that the failure recovery process for the camera is successful.
[0087] In some embodiments, before performing the power-on process on the camera, the processing module 501 is further configured to:
[0088] When performing the power-off process on the camera, set the current state of the camera to a transition state, where the transition state is used to indicate that the camera will be powered on after waiting for a preset second duration; and / or,
[0089] After performing the power-off process on the camera, set the current state of the camera to a power-on state, where the power-on state is used to indicate that the camera will be powered on.
[0090] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.
[0091] The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the camera fault detection method provided by the present disclosure are implemented.
[0092] Figure 6 FIG. is a schematic structural diagram of a terminal device shown according to an exemplary embodiment. For example, the terminal device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or other terminal devices.
[0093] Referring to Figure 6 , the terminal device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an input / output interface 612, a sensor component 614, and a communication component 616.
[0094] The processing component 602 generally controls the overall operation of the device 600, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-described camera fault detection method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.
[0095] The memory 604 is configured to store various types of data to support the operation of the device 600. Examples of such data include instructions for any application or method operating on the device 600, contact data, phone book data, messages, pictures, videos, and the like. The memory 604 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.
[0096] The power component 606 provides power to various components of the device 600. The power component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 600.
[0097] The multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the terminal device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.
[0098] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC) that is configured to receive external audio signals when the device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 604 or transmitted via the communication component 616. In some embodiments, the audio component 610 further includes a speaker for outputting audio signals.
[0099] The input / output interface 612 provides an interface between the processing component 602 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a start button, and a lock button.
[0100] The sensor component 614 includes one or more sensors for providing an assessment of various aspects of the state of the device 600. For example, the sensor component 614 can detect the on / off state of the device 600, the relative positioning of components, such as the display and keypad of the device 600, the sensor component 614 can also detect a change in the position of the device 600 or a component of the device 600, the presence or absence of user contact with the device 600, the orientation or acceleration / deceleration of the device 600, and the temperature change of the device 600. The sensor component 614 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 614 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 614 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0101] The communication component 616 is configured to facilitate communication between the device 600 and other devices in a wired or wireless manner. The device 600 can access a wireless network based on communication standards, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0102] In an exemplary embodiment, the device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described camera fault detection method.
[0103] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions. The above instructions can be executed by a processor 620 of the device 600 to complete the above-described upper camera fault detection method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0104] In addition to being an independent electronic device, the above device may also be a part of an independent electronic device. For example, in one embodiment, the device may be an integrated circuit (IC) or a chip. The integrated circuit may be one IC or a collection of multiple ICs; the chip may include, but is not limited to, the following types: GPU (Graphics Processing Unit, graphics processor), CPU (Central Processing Unit, central processor), FPGA (Field Programmable Gate Array, programmable logic array), DSP (Digital Signal Processor, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), SOC (System on Chip, SoC, system-on-chip or system-level chip), etc. The above integrated circuit or chip may be used to execute executable instructions (or code) to implement the above-described camera fault detection method. The executable instructions may be stored in the integrated circuit or chip, or may be obtained from other devices or equipment. For example, the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instructions may be stored in the memory, and when the executable instructions are executed by the processor, the above-described camera fault detection method is implemented; or, the integrated circuit or chip may receive the executable instructions through the interface and transmit them to the processor for execution to implement the above-described camera fault detection method.
[0105] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-described camera fault detection method when executed by the programmable device.
[0106] Please refer to Figure 7 FIG. is a schematic structural diagram of a chip shown according to an exemplary embodiment. As Figure 7 shown, the chip 700 includes a processor 701 and an interface 702. Optionally, a memory 703 may further be included. Among them, the number of processors 701 may be one or more, and the number of interfaces 702 may be multiple.
[0107] In one embodiment, for the case where the chip is used to implement the method embodiments of the present disclosure:
[0108] The interface 702 is used to receive or output signals;
[0109] The processor 701 is used to execute some or all of the content in the camera fault detection method embodiments.
[0110] It can be understood that the processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method embodiments may be completed by the integrated logic circuit in the hardware of the processor or instructions in software form. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0111] Understandably, the memory in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.
[0112] It should be pointed out here that the descriptions of the above storage medium, device, and chip embodiments are similar to those of the above method embodiments and have similar beneficial effects to the method embodiments. For the technical details not disclosed in the storage medium, storage medium, and device embodiments of the present disclosure, please refer to the descriptions of the method embodiments of the present disclosure for understanding.
[0113] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only to be considered exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.
[0114] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.
Claims
1. A camera fault detection method, characterized in that: The camera includes a fault detection component, and the method includes: Performing fault detection on the camera based on the fault detection component to obtain fault information of the camera; Among them, when the fault detection component includes a camera driver component, the fault information includes at least one of a deserializer hardware failure, a camera outflow failure and a camera frame failure; when the fault detection component includes a register component, the fault information includes at least one of a camera module failure, a deserializer communication failure, a serializer failure, a power supply chip failure and a storage chip failure.
2. The method according to claim 1, characterized in that If the fault detection component includes the camera driving component, then the performing fault detection on the camera based on the fault detection component to obtain the fault information of the camera includes at least one of the following: Calling a first program class in the camera driver component to perform abnormal monitoring on a deserializer in the camera to capture a hardware failure of the deserializer of the camera; Calling a second program class in the camera driver component to detect whether a data stream of the camera is received within a preset first time period to determine whether a camera outflow failure occurs; Calling a third program class in the camera driver component to detect whether the camera frame failure occurs; When the data stream of the camera is not received within the preset first time period, it is determined that the camera outflow failure occurs, and the camera frame failure includes at least one of the following failures: frame loss, non-continuous frame, frame capture processing ICP timeout and frame format error.
3. The method according to claim 1, characterized in that If the fault detection component includes the register component, then performing fault detection on the camera based on the fault detection component to obtain fault information of the camera includes: Based on the handle corresponding to the register component, access the address of the register component and read the register status of the register component, wherein the register status is at least used to reflect whether the camera fails; Based on the registration status of the register component, the fault information of the camera is obtained.
4. The method according to claim 3, characterized in that The obtaining the fault information of the camera based on the storage state of the register component includes: When the registration state of the register component is not a preset state, it is determined that the camera fails, and the failure information of the camera is obtained.
5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the number of consecutive occurrences of the fault information is greater than or equal to a preset number, the fault information is reported.
6. The method according to any one of claims 1 to 5, characterized in that When the cameras within N camera ranges are connected to N deserializers, and each deserializer is connected to M device control chips, the deserializer communication failures include N×M, wherein each deserializer communication failure is used to indicate that a communication failure occurs in the data stream of the camera within the corresponding camera range, and N and M are both positive integers.
7. The method according to any one of claims 1 to 5, characterized in that The power supply chip failure includes any one of the following failures: open circuit, short circuit, overvoltage and undervoltage.
8. The method according to any one of claims 1 to 5, characterized in that The method further comprises: After the camera fails, a failure recovery process is performed on the camera based on a setting state of a finite state machine, where the setting state is used to indicate a current state of the camera.
9. The method according to claim 8, characterized in that The performing fault recovery processing on the camera based on the setting state of the finite state machine includes: After the camera fails, setting the current state of the camera to an unavailable state and powering off the camera; After the power-off process, powering on the camera; After the camera is successfully powered on, the current state of the camera is set to an available state to indicate that the fault recovery process for the camera is successful.
10. The method according to claim 9, characterized in that Before powering on the camera, the method further includes: When the camera is powered off, the current state of the camera is set to a transition state, where the transition state is used to indicate that the camera is powered on after waiting for a preset second time period; and / or, After the camera is powered off, the current state of the camera is set to a power-on state, where the power-on state is used to indicate that the camera is powered on.
11. A camera fault detection device, characterized in that: The camera includes a fault detection component, and the device includes: a processing module, configured to perform fault detection on the camera based on the fault detection component to obtain fault information of the camera; Among them, when the fault detection component includes a camera driver component, the fault information includes at least one of a deserializer hardware failure, a camera outflow failure and a camera frame failure; when the fault detection component includes a register component, the fault information includes at least one of a camera module failure, a deserializer communication failure, a serializer failure, a power supply chip failure and a storage chip failure.
12. A terminal device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions to implement the steps of the method according to any one of claims 1 to 10.