Fault detection method, system, equipment, vehicle and product for image acquisition device

Abstract

The present disclosure relates to a fault detection method, system, device, vehicle and product for an image acquisition device, and relates to the field of vehicle detection technology. The fault detection method is applied to a vehicle, and the image acquisition device is connected to the vehicle in communication. The present disclosure performs software fault detection on the image acquisition device according to image embedded information to obtain a fault detection result of the image acquisition device; the image embedded information is used to describe additional data information of the original image frame. The present disclosure can improve the accuracy and efficiency of the image acquisition device.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle detection technology, and in particular to a fault detection method, system, device, vehicle, computer-readable storage medium and computer program product for an image acquisition device. Background Art

[0002] With the development of automobile intelligence and informatization, intelligent driving technology continues to advance, and the domain controller of the intelligent driving domain has become one of the key components. The domain controller can simulate the driver's operating behavior and decision-making through the image data collected by the image acquisition device to improve the driving perception, planning and control capabilities.

[0003] However, when a potential fault occurs in the image acquisition device, the current fault detection method requires uploading the complete image data and working status data to the server, or using electrical inspection equipment to perform a drive scan when the vehicle is stationary, resulting in a certain lag in the fault detection of the image acquisition device and low detection efficiency. In addition, the above fault detection methods are also difficult to ensure the accuracy of fault location, affecting the normal operation of the intelligent driving system. Summary of the invention

[0004] The embodiments of the present disclosure provide a fault detection method for an image acquisition device, which improves the accuracy and efficiency of fault detection for the image acquisition device, so as to at least partially solve the above technical problems.

[0005] In order to achieve the above object, according to a first aspect of the present disclosure, a fault detection method of an image acquisition device is provided, which is applied to a vehicle, wherein the image acquisition device is communicatively connected to the vehicle, and the method comprises:

[0006] The image acquisition device is subjected to software fault detection according to the image embedded information to obtain a fault detection result of the image acquisition device; the image embedded information is used to describe additional data information of the original image frame.

[0007] Optionally, the image embedding information is used to describe additional data information of the original image frame, including:

[0008] Sending each image frame acquired by the image acquisition device to an image acquisition device access service component of the vehicle;

[0009] The image embedding information is inserted into each image frame through a software development kit in a service component accessed by the image acquisition device.

[0010] Optionally, performing software fault detection on the image acquisition device according to the image embedded information includes:

[0011] Each of the image frames is sent to a domain controller, and the domain controller performs software fault detection on the image acquisition device according to the image embedding information received in the image frame.

[0012] Optionally, the image embedded information at least includes a frame sequence number, a frame rate and a frame delay of the image frame; and performing software fault detection on the image acquisition device according to the image embedded information includes:

[0013] A first software fault detection is performed according to the frame sequence number of the image frame, and a second software fault detection is performed according to the frame rate and frame delay of the image frame.

[0014] Optionally, performing a first software fault detection according to a frame sequence number of the image frame, and performing a second software fault detection according to a frame rate and a frame delay of the image frame, includes:

[0015] Determining whether there is frame loss in the image frame according to the frame sequence number of the image frame, and when there is frame loss, determining whether there is a software fault in the vehicle image acquisition device according to the number of lost image frames;

[0016] And / or, judging whether the image frame has any jamming based on the frame rate and frame delay of the image frame; if there is any jamming, judging whether the vehicle image acquisition device has any software fault based on the total duration of the jamming.

[0017] Optionally, the method further comprises:

[0018] Acquiring hardware operating status information of the image acquisition device;

[0019] A hardware fault detection is performed on the image acquisition device according to the hardware operation status information.

[0020] Optionally, the hardware operating status information includes an operating temperature and / or an operating voltage of the image acquisition device.

[0021] Optionally, performing hardware fault detection on the image acquisition device according to the hardware operation status information includes:

[0022] When the operating temperature is outside the preset temperature range, determining that the image acquisition device has hardware abnormality information indicating a temperature abnormality;

[0023] And / or, when the operating voltage is outside the preset voltage range, it is determined that hardware abnormality information indicating voltage abnormality exists in the image acquisition device.

[0024] Optionally, the method further comprises:

[0025] When the duration of the hardware abnormality information is greater than a preset duration threshold, a hardware fault warning message indicating that a hardware fault exists in the image acquisition device is issued.

[0026] Optionally, the method further comprises:

[0027] Performing periodic software fault detection on the image acquisition device to obtain software fault information;

[0028] When the data value of the software fault information is restored to a first preset normal range, software fault recovery information indicating that the software fault of the image acquisition device is resolved is issued.

[0029] Optionally, the method further comprises:

[0030] Performing periodic hardware fault detection on the image acquisition device to obtain hardware fault information;

[0031] When the data value of the hardware fault information is restored to a second preset normal range, hardware fault recovery information indicating that the hardware fault of the image acquisition device is resolved is issued.

[0032] According to a second aspect of the present disclosure, there is also provided a fault detection system for an image acquisition device, which is applied to a vehicle, and the system comprises an image acquisition device, a software development kit and a domain controller; the image acquisition device is communicatively connected to the vehicle via a bidirectional two-wire synchronous serial bus; wherein:

[0033] The image acquisition device is used to acquire image data;

[0034] The software development kit is used to insert image embedding information into the image data;

[0035] The domain controller is used to perform software fault detection on the image acquisition device according to the image embedding information.

[0036] Optionally, the system further includes a bidirectional two-wire synchronous serial bus and / or a microcontroller unit, which are used to read hardware operating status information of the image acquisition device; the hardware operating status information includes the operating temperature and / or operating voltage of the image acquisition device.

[0037] According to a third aspect of the present disclosure, a computer-readable storage medium is further provided, wherein program instructions are stored in the computer-readable storage medium, and the program instructions implement the method described above when executed.

[0038] According to a fourth aspect of the present disclosure, a computer program product is also provided, wherein the computer program product comprises program instructions, and when the program instructions are executed by a processor, the method as described above is implemented.

[0039] According to a fifth aspect of the present disclosure, an electronic device is also provided for executing the method as described above.

[0040] According to a sixth aspect of the present disclosure, a vehicle is also provided, comprising the electronic device as described above, or executing the method as described above.

[0041] In the fault detection method for an image acquisition device of an embodiment of the present disclosure, by acquiring the image embedded information of the image acquisition device in real time and combining it with a variety of additional information in the image embedded information, targeted software fault detection is performed on the image acquisition device, thereby being able to comprehensively and accurately detect the fault location and cause of the image acquisition device, thereby improving fault detection efficiency and enhancing user driving safety and experience.

[0042] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0044] In order to more completely understand the present disclosure and its beneficial effects, the following description will be made in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0045] Figure 1 It is a schematic diagram of the existing fault detection of the image acquisition device;

[0046] Figure 2 is a flowchart of a fault detection method for an image acquisition device provided in an exemplary embodiment of the present disclosure;

[0047] Figure 3 is a schematic diagram of a domain controller performing fault detection on an image acquisition device provided in an exemplary embodiment of the present disclosure;

[0048] Figure 4 is a schematic diagram of an image acquisition device access service component performing fault detection on an image acquisition device provided in an exemplary embodiment of the present disclosure;

[0049] Figure 5 is a schematic diagram of processing an image frame provided in an exemplary embodiment of the present disclosure;

[0050] Figure 6 is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure;

[0051] Figure 7 is a block diagram of a vehicle provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] Nowadays, there are two main ways to detect faults in image acquisition devices, such as Figure 1 As shown, one method is to upload the working status data of the device to be detected in the historical period and the complete image data collected by it to the server, and analyze the above received data through the image acquisition device access service component, image use service component and various sub-service components of the server, and then give the corresponding fault detection result. Fault detection has a certain lag, the detection efficiency is extremely low, and it is impossible to guarantee accurate fault detection results.

[0054] Another detection method is to use electrical inspection equipment. The vehicle needs to be stationary and the image acquisition device needs to be scanned by the electrical inspection equipment to capture and analyze the image data and working status of the image acquisition device. This method also has the defects of lag and low detection efficiency, and cannot achieve real-time and rapid fault location and fault analysis.

[0055] Based on the above problems, the present disclosure provides a fault detection method for an image acquisition device, which obtains the image embedded information of the image acquisition device in real time and performs targeted software fault detection on the image acquisition device in combination with a variety of additional information in the image embedded information, thereby being able to comprehensively and accurately detect the fault location and cause of the image acquisition device, improve fault detection efficiency, and enhance user driving safety and experience.

[0056] The present disclosure is described below in conjunction with specific embodiments.

[0057] Figure 2 is a schematic diagram of a fault detection method for an image acquisition device provided by an embodiment of the present disclosure, such as Figure 2 As shown, the method can be applied to a vehicle, the image acquisition device is communicatively connected with the vehicle, and the method can include the following steps:

[0058] Step S101: Perform software fault detection on the image acquisition device according to the image embedding information to obtain a fault detection result of the image acquisition device.

[0059] In the present disclosure, the image acquisition device may be a device installed on a vehicle for capturing surrounding images, such as a camera, a driving recorder, an image sensor, etc. In some embodiments, the image acquisition device may be applied to a driving assistance system and some vehicle driving safety scenarios. For example, the vehicle may use the image acquisition device to help the driver identify road conditions, lane markings, traffic signs, etc. during driving. For example, the image acquisition device may provide environmental perception data for an autonomous driving system to support the vehicle's automatic navigation and obstacle avoidance functions.

[0060] In some embodiments, the image acquisition device can generate continuous image frames during the process of capturing images, and the generated image frames can be transmitted through the MIPI protocol to achieve efficient and stable image frame transmission through the MIPI protocol.

[0061] The image embedded information is a kind of additional description information used to describe the additional data information of the original image frame. Accordingly, inserting the image embedded information into the image frame can be understood as adding additional description information into the image frame.

[0062] In some embodiments, in some embodiments, the image embedded information may include at least the frame number, frame rate and frame delay of the image frame. Among them, the frame number refers to a unique number used to identify each image frame, which is used to track and identify the frame sequence. The frame rate refers to the number of frames transmitted per second, which is used to evaluate the smoothness and transmission performance of the video stream. Frame delay refers to the time required for an image frame to be acquired and transmitted to the receiving end. After clarifying the above information, as an example only, the specific content of an image embedded information can be expressed as "[image embedded information: frame number = 1234, frame rate = 30fps, delay = 10ms]".

[0063] In some embodiments, image embedding information may be inserted into specific rows of original pixels of each image frame, for example, one or more rows of image embedding information may be inserted into the top or bottom of each frame. It should be noted that the image frame after the image embedding information is inserted does not interfere with normal image display and processing.

[0064] In order to better implement the method of the embodiment of the present disclosure, in a possible example, the image embedding information used to describe the additional data information of the original image frame may include:

[0065] Sending each of the image frames captured by the image capture device to an image capture device access service component of the vehicle;

[0066] The image embedding information is inserted into each image frame through a software development kit in a service component accessed by the image acquisition device.

[0067] Among them, the image acquisition device access service component is a component of the domain controller, which is used to manage and process data such as image frames and hardware operating status information of the image acquisition device, and perform abnormal analysis on the above data.

[0068] like Figure 4 As shown, when the image acquisition device access service component detects a software failure in the image acquisition device, it can first access the relevant information obtained by the serializer / deserializer through the I2C bus, and in this way perform initialization operations on the image acquisition device when it is subsequently started. For example, the power management information, device self-test information, and status report information of the image acquisition device can be initialized.

[0069] Among them, the serializer / deserializer includes two components, a serializer and a deserializer. The serializer can be used to convert parallel data into serial data, and the deserializer can convert serial data back into parallel data. Accessing and reading the data of the image acquisition device through the serializer / deserializer can ensure high speed and reliability of data transmission.

[0070] In some embodiments, after the image acquisition device is started and initialized, it can start image acquisition, capture images around the vehicle, and generate continuous image frames, each image frame containing visual information around the vehicle.

[0071] In some embodiments, the image acquisition device access service component can convert each image frame into a serial data stream through a serializer so that each image frame can be subsequently transmitted on a high-speed transmission bus, for example, through the MIPI protocol, and then restored to parallel image frame data through a deserializer.

[0072] In some embodiments, the software development kit may be NVSDK (NVIDIA Software Development Kit). NVSDK is a software toolkit for processing image data, which can provide rich interfaces and functions, and support the parsing, processing and enhancement of image data. In some embodiments, after receiving each frame of image data, the software development kit can first parse the image frame to obtain basic information of the image frame. Then, according to preset rules, image embedding information including information such as frame number, timestamp, frame rate and frame delay can be inserted into a specific row of the image frame. The inserted image embedding information can be located at the top or bottom of the image frame to ensure that it does not affect the normal display and processing of the image content.

[0073] In some embodiments, in addition to the frame number, frame rate, and frame delay, the software development kit may also insert timestamp information into the image frame. For example, the image embedded information inserted into the image frame may be “[frame number=0015, timestamp=2024-06-13 10:00:15.250, frame rate=30fps, frame delay=12ms]”.

[0074] In the above manner, regardless of whether the vehicle is in a driving state or a stationary state, image embedded information can be inserted into the image frame of the image acquisition device in real time and stably, so that real-time, frame-by-frame fault detection operations can be performed on the image frame later, thereby improving the stability, timeliness and accuracy of fault analysis of the image acquisition device.

[0075] In some embodiments, software fault detection may be performed on the image acquisition device according to the image embedding information to obtain a fault detection result of the image acquisition device.

[0076] It can be understood that the hardware operation status information of the image acquisition device can reflect the working status of the image acquisition device, and the image embedded information in the image frame acquired by the image acquisition device is additional description information of the image frame. Figure 3 and Figure 4 As shown, the present disclosure can perform hardware fault detection by analyzing the hardware operation status information of the image acquisition device, such as whether the temperature or voltage of the image acquisition device is abnormal. Software fault detection can also be performed by analyzing the image embedded information of the image frame of the image acquisition device, such as whether the image acquired by the image acquisition device has frame loss or freeze.

[0077] In order to better implement the method of the embodiment of the present disclosure, in a possible example, the software fault detection in step S103 can be implemented in the following manner:

[0078] Each of the image frames is sent to a domain controller, and the domain controller performs software fault detection on the image acquisition device according to the image embedding information received in the image frame.

[0079] In some embodiments, the domain controller may include an image acquisition device access service component and an image usage service component. Among them, the image acquisition device access service component can perform preliminary fault detection analysis, for example, it can detect whether there is abnormal data in the image acquisition device, and the abnormal data can be abnormal temperature, abnormal data frame, etc. The image usage service component can be combined with relevant algorithms, and further analyzed in combination with the preliminary fault detection results of the image acquisition device access service component to obtain the final fault detection result, and determine whether to issue a fault report or fault alarm to the user, for example, the alarm information of the image acquisition device failure can be displayed on the central control screen of the vehicle.

[0080] In some embodiments, software fault detection of the image acquisition device may be performed in the following manner:

[0081] A first software fault detection is performed according to the frame sequence number of the image frame, and a second software fault detection is performed according to the frame rate and frame delay of the image frame.

[0082] In some embodiments, the present disclosure can perform different types of software fault detection on the image acquisition device through the frame sequence number, frame rate and frame delay of the image frame, thereby ensuring the comprehensiveness of the software fault detection on the image acquisition device.

[0083] In some embodiments, the first software fault detection and the second software fault detection may be performed on the image acquisition device in the following manner:

[0084] Determining whether there is frame loss in the image frame according to the frame sequence number of the image frame, and when there is frame loss, determining whether there is a software fault in the vehicle image acquisition device according to the number of lost image frames;

[0085] And / or, judging whether the image frame has any jamming based on the frame rate and frame delay of the image frame; if there is any jamming, judging whether the vehicle image acquisition device has any software fault based on the total duration of the jamming.

[0086] like Figure 5 As shown, in some embodiments, after the software development kit inserts the image embedded information into the image frame, the image acquisition device access service component can receive the processed image frame, and separate the effective pixels and the image embedded information from the processed image frame, thereby extracting the relevant image embedded information, such as the frame number, frame rate and exposure time, etc., and then preprocessing and reassembling the extracted image embedded information to obtain assembled meta data, that is, metadata of the image embedded information. As an example only, the metadata of the image embedded information can be expressed as:

[0087] struct MetaData

[0088] {

[0089] unit64_t data 1; (timestamp of each frame of data received)

[0090] unit64_t data2; (timestamp of the complete image moment)

[0091] unit64_t data 3; (image center row exposure center point timestamp)

[0092] unit64_t data 4; (image frame number)

[0093] unit64_t data 5; (exposure gain)

[0094] unit64_t data 6; (fault status)

[0095] };

[0096] In some embodiments, the image acquisition device access service component can further perform software fault detection on the image acquisition device by analyzing the metadata of the image embedded information. Specifically, assuming that the frame rate of the image acquisition device is set to 30FPS, indicating that 30 image frames can be acquired per second, when the metadata of the image embedded information is analyzed, it is found that the image acquisition device only acquires 29 image frames in a certain second, for example, the image frames with frame numbers 1 to 29 are complete, and the image frame number 30 is missing, then it can be detected that there is currently a frame loss situation, and the image frame number 30 can be first determined as Figure 4 The software abnormal data in , that is, the abnormal data frame.

[0097] Furthermore, the image acquisition device access service component can send the detected software anomaly data to the corresponding image usage service component, and the image usage service component can determine the number of image frames of the software anomaly data. For example, the number of lost frames can be further detected. When the number of lost frames is greater than the frame loss threshold, for example, the number of lost frames per second is greater than 5 frames, it is determined that a software fault of frame loss has been detected in the vehicle.

[0098] Similar to the detection process of frame loss failure, when the frame delay of an image frame exceeds the single frame duration threshold, for example, exceeds 50ms, the image frame can be determined as having excessive frame delay, and the image frame will be stuck. The image acquisition access service component can determine the image frame as software abnormal data.

[0099] Furthermore, the image acquisition access service component can send the image frame in which the software abnormal data is detected to the image use service component, and the image use service component can determine whether the vehicle image acquisition device has a software fault according to the total duration of the jam. For example, when the total duration of the jam exceeds 2s, it is determined that the image acquisition device has a jam fault with excessive frame delay.

[0100] It can be understood that no matter it is a software failure such as frame loss or image freeze, for an instantaneous and short-term software failure, such as only losing one image frame, or a frame delay of only 1ms, it will not have a significant impact on the image acquisition function of the image acquisition device. There is no need to determine the appearance of the above software abnormal data as an actual failure of the image acquisition device, and there is no need to perform corresponding early warning and reporting operations to avoid unnecessary impact on user driving.

[0101] In some embodiments, the method of the present disclosure may further include:

[0102] Acquiring hardware operating status information of the image acquisition device;

[0103] A hardware fault detection is performed on the image acquisition device according to the hardware operation status information.

[0104] The hardware operation status information refers to various data reflecting the current working status of the hardware. For example, the hardware operation status information may include information such as the working temperature and working voltage of the image acquisition device.

[0105] In some embodiments, the serializer (SerDes) / deserializer (DesSer) can be accessed through the I2C bus to read the hardware operation status information of the image acquisition device. It can be understood that since the I2C bus is usually used to transmit data with a small amount and a low acquisition frequency, the transmission capacity of the I2C bus is suitable for obtaining the status information of the image acquisition device.

[0106] In some embodiments, the image acquisition device is communicatively connected to the vehicle via a bidirectional two-wire synchronous serial bus.

[0107] In order to better implement the method of the embodiment of the present disclosure, in a possible example, the hardware operating status information includes the operating temperature and operating voltage of the image acquisition device.

[0108] In some embodiments, when the operating temperature is outside the preset temperature range, hardware abnormality information indicating that the image acquisition device has a temperature abnormality is determined; and / or, when the operating voltage is outside the preset voltage range, hardware abnormality information indicating that the image acquisition device has a voltage abnormality is determined.

[0109] In some embodiments, when the duration of the hardware abnormality information is greater than a preset duration threshold, a hardware fault warning message indicating that a hardware fault exists in the image acquisition device is issued.

[0110] like Figure 3 and Figure 4 As shown, the present disclosure can synchronously read the hardware operation status information in the register of the image acquisition device through the I2C bus and the MCU to realize multi-interface interactive mode reading. In this way, the data interaction efficiency can be improved through the two reading channels of the I2C bus and the MCU, and the format and type of the data can be enriched.

[0111] In a specific implementation, the current I2C state can be read through the I2C bus to obtain the state of the image acquisition device and hardware fault information in real time, such as hardware fault information such as operating voltage and operating temperature. It can be understood that since the hardware fault information can change the content of the register address of the image acquisition device, the image acquisition device access service component can periodically read the I2C address content to obtain the corresponding hardware abnormality data. In some embodiments, in addition to reading the I2C state of the image acquisition device, different state information can be written by calling the I2C bus to achieve the regulation of the image acquisition device.

[0112] In some embodiments, it is possible to compare whether the working voltage is within a preset voltage range and whether the working temperature is within a preset temperature range to determine whether the image acquisition device has hardware abnormality information. For example, assuming that the preset temperature range is -20°C to 85°C and the preset voltage range is 3.0V to 3.6V, and in a certain reading, the obtained working temperature is 90°C and the working voltage is 3.3V, then it can be determined that the working temperature of 90°C exceeds the preset temperature range (-20°C to 85°C), and is determined as hardware abnormality information of abnormal temperature, and the working voltage of 3.3V is within the preset voltage range (3.0V to 3.6V), and is determined as normal voltage.

[0113] In some embodiments, when the duration of the hardware abnormality information is longer than a preset duration threshold, a hardware fault warning information indicating that the image acquisition device has a hardware fault is issued. For example, assuming that the preset duration threshold is 10 seconds, it can be understood that when the duration of the hardware abnormality information in the above example that the operating temperature reaches 90°C is longer than 10 seconds, a fault warning information indicating that the image acquisition device has a hardware fault is issued.

[0114] It can be understood that, similar to software fault detection, for instantaneous and short-term hardware faults, such as abnormalities in the working voltage and working temperature of the image acquisition device that only last for a few seconds, it will not have a significant impact on the hardware function of the image acquisition device. There is no need to determine the appearance of the above hardware abnormality information as an actual fault in the image acquisition device, and there is no need to perform corresponding early warning and reporting operations, so as to avoid unnecessary impact on user driving.

[0115] In some embodiments, the method of the present disclosure may further include:

[0116] Performing periodic software fault detection on the image acquisition device to obtain software fault information;

[0117] When the data value of the software fault information is restored to a first preset normal range, software fault recovery information indicating that the software fault of the image acquisition device is resolved is issued.

[0118] In some embodiments, the image acquisition device may be periodically tested for software faults in the above manner to obtain software fault information. For example, software fault information such as frame numbers of frame loss and frame freezes may be obtained. In some embodiments, when the data value of the software fault information (such as frame rate, frame delay, etc.) is restored to a preset normal range, it is considered that the software fault of the image acquisition device has been resolved.

[0119] In some embodiments, the method of the present disclosure may further include:

[0120] Performing periodic hardware fault detection on the image acquisition device to obtain hardware fault information;

[0121] When the data value of the hardware fault information is restored to a second preset normal range, hardware fault recovery information indicating that the hardware fault of the image acquisition device is resolved is issued.

[0122] In some embodiments, the image acquisition device may be periodically tested for hardware faults in the above manner to obtain hardware fault information. For example, the operating temperature and operating voltage of the image acquisition device may be periodically read through the I2C bus and the MCU to determine hardware fault information such as abnormal operating temperature and abnormal operating voltage. In some embodiments, when the data value (such as temperature, voltage, etc.) of the hardware fault information is restored to a preset normal range, the system considers that the hardware fault has been resolved and issues fault recovery information for the image acquisition device.

[0123] It should be noted that the present invention can simultaneously and periodically detect various software fault information (such as frame loss, freeze) and hardware fault information (such as temperature abnormality, voltage abnormality) of the vehicle-mounted image acquisition device, and compare this information with the values ​​within the preset normal range. When all fault information is restored to the values ​​within the preset normal range and remains stable, the system can issue a fault recovery message indicating that the fault has been resolved.

[0124] Through the fault detection system for the image acquisition device provided by the embodiment of the present disclosure, through the above-mentioned technical scheme, by acquiring the hardware operation status information of the image acquisition device and the image embedded information of the image frame in real time, and combining the above information to perform targeted fault detection on the image acquisition device, distinguish between software faults and hardware faults, so as to comprehensively and accurately detect the fault location and cause of the image acquisition device, improve fault detection efficiency, and enhance the user's driving safety and experience.

[0125] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0126] In some embodiments, the present disclosure further provides a fault detection system for an image acquisition device, which is applied to a vehicle. In some embodiments, the fault detection system for an image acquisition device may include an image acquisition device, a bidirectional two-wire synchronous serial bus, a microcontroller unit, a software development kit, and a domain controller. The image acquisition device is connected to the vehicle through a bidirectional two-wire synchronous serial bus.

[0127] In the system: an image acquisition device is used to acquire image data; a bidirectional two-wire synchronous serial bus and a microcontroller unit are used to read hardware operating status information of the image acquisition device; the hardware operating status information includes the operating temperature and / or the operating voltage of the image acquisition device; a software development kit is used to insert image embedding information into the image data; and a domain controller is used to perform software fault detection on the image acquisition device according to the image embedding information.

[0128] Figure 6 FIG. 1 is a block diagram of an electronic device 300 according to an exemplary embodiment. Figure 6 As shown, the electronic device 300 may include: a processor 301 and a memory 302. The electronic device 300 may also include one or more of a multimedia component 303, an input / output (I / O) component 304, and a communication component 305.

[0129] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned fault detection method of the image acquisition device. The memory 302 is used to store various types of data to support the operation of the electronic device 300, and these data may include, for example, instructions for any application or method used to operate on the electronic device 300, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 302 or sent through the communication component 305. The audio component also includes at least one speaker for outputting audio signals. The I / O component 304 provides an interface between the processor 301 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 305 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0130] In an exemplary embodiment, the electronic device 300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned fault detection method of the image acquisition device.

[0131] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the domain controller, the steps of the above-mentioned method for detecting a fault in an image acquisition device are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 302 including program instructions, and the above-mentioned program instructions may be executed by the processor 301 of the electronic device 300 to complete the following steps:

[0132] The image acquisition device is subjected to software fault detection according to the image embedded information to obtain a fault detection result of the image acquisition device; the image embedded information is used to describe additional data information of the original image frame.

[0133] Figure 7 is a block diagram of a vehicle provided in an embodiment of the present disclosure, such as Figure 7 As shown, the vehicle 400 includes the electronic device 300 described above.

[0134] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the charging control methods recorded in the above method embodiments.

[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0140] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable storage unit. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage unit, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage unit includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0141] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage unit, which can include: a flash drive, a read-only storage unit, a random access memory, a magnetic disk or an optical disk, etc.

[0142] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0143] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0144] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A fault detection method for an image acquisition device, applied to a vehicle, wherein the image acquisition device is communicatively connected to the vehicle, characterized in that: The method comprises: performing software fault detection on the image acquisition device according to the image embedded information carried by the image frame to obtain a software fault detection result of the image acquisition device, and performing hardware fault detection on the image acquisition device according to the hardware operation state information of the image acquisition device to obtain a hardware fault detection result of the image acquisition device; the image embedded information is used to describe additional data information of the original image frame; The performing software fault detection on the image acquisition device according to the image embedded information carried by the image frame to obtain the software fault detection result of the image acquisition device includes: The software development kit inserts the image embedding information into the original image frame to obtain the image frame, the image acquisition device access service component of the domain controller of the vehicle receives the image frame obtained after the image embedding information is inserted, separates the effective pixels and the image embedding information from the received image frame, pre-processes and reassembles the separated image embedding information to obtain metadata of the image embedding information, and performs software fault detection on the image acquisition device based on the metadata; wherein the image embedding information includes frame rate and frame delay; The image usage service component of the domain controller further analyzes the result of software fault detection on the image acquisition device access service component to obtain the software fault detection result of the image acquisition device.

2. The fault detection method of the image acquisition device according to claim 1, characterized in that: The image embedding information also includes a frame sequence number of the image frame; The image acquisition device access service component performs software fault detection on the image acquisition device, including: A first software fault detection is performed according to the frame rate and the frame sequence number of the image frame, and a second software fault detection is performed according to the frame delay of the image frame.

3. The fault detection method of the image acquisition device according to claim 2, characterized in that: The performing a first software fault detection according to the frame rate and the frame sequence number of the image frame includes: Determining whether there is frame loss in the image frame according to the frame rate and the frame sequence number of the image frame; When frame loss occurs, the image usage service component of the domain controller further analyzes the result of software fault detection on the image acquisition device access service component to obtain the software fault detection result of the image acquisition device, including: the image usage service component determines the number of lost image frames, and determines whether there is a software fault in the image acquisition device based on the number.

4. The method according to claim 2, characterized in that: The performing a second software fault detection according to the frame delay of the image frame comprises: Determining whether there is a freeze in the image frame according to the frame delay of the image frame; When there is a freeze, the image using service component of the domain controller further analyzes the result of software fault detection on the image acquisition device access service component to obtain the software fault detection result of the image acquisition device, including: According to the total duration of the freeze, determine whether there is a software fault in the image acquisition device.

5. The method according to claim 1, characterized in that The hardware operation status information includes the operating temperature and / or operating voltage of the image acquisition device.

6. The method according to claim 5, characterized in that The performing hardware fault detection on the image acquisition device according to the hardware operation status information comprises: When the operating temperature is outside the preset temperature range, determining that the image acquisition device has hardware abnormality information indicating a temperature abnormality; And / or, when the operating voltage is outside a preset voltage range, it is determined that hardware abnormality information indicating a voltage abnormality exists in the image acquisition device.

7. The method according to claim 6, characterized in that The method further comprises: When the duration of the hardware abnormality information is greater than a preset duration threshold, a hardware fault warning message indicating that a hardware fault exists in the image acquisition device is issued.

8. The fault detection method of the image acquisition device according to claim 1, characterized in that: The performing software fault detection on the image acquisition device according to the image embedded information carried by the image frame to obtain the software fault detection result of the image acquisition device includes: Performing periodic software fault detection on the image acquisition device according to the image embedded information carried by the image frame to obtain a software fault result of the image acquisition device; When the data value of the software fault result is restored to a first preset normal range, software fault recovery information indicating that the software fault of the image acquisition device has been resolved is issued.

9. The method according to any one of claims 1 to 4, characterized in that: The performing hardware fault detection on the image acquisition device according to the hardware operation status information of the image acquisition device to obtain the hardware fault detection result of the image acquisition device includes: According to the hardware operation status information of the image acquisition device, periodically perform hardware fault detection on the image acquisition device to obtain a hardware fault result; When the data value of the hardware fault result is restored to a second preset normal range, hardware fault recovery information indicating that the hardware fault of the image acquisition device is resolved is issued.

10. A fault detection system for an image acquisition device, applied to a vehicle, characterized in that: The system includes an image acquisition device, a software development kit and a domain controller; the image acquisition device is connected to the vehicle through a bidirectional two-wire synchronous serial bus; wherein: The image acquisition device is used to acquire original image frames; The software development kit is used to insert image embedding information into the original image frame to obtain an image frame carrying the image embedding information, wherein the image embedding information is used to describe additional data information of the original image frame; The domain controller includes an image acquisition device access service component and an image use service component, wherein the image acquisition device access service component receives the image frame obtained after inserting the image embedding information, separates the effective pixels and the image embedding information from the received image frame, pre-processes and reassembles the separated image embedding information to obtain metadata of the image embedding information, and performs software fault detection on the image acquisition device based on the metadata; wherein the image embedding information includes a frame rate and a frame delay; The image usage service component of the domain controller further analyzes the result of software fault detection on the image acquisition device access service component to obtain the software fault detection result of the image acquisition device.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, which implement the method according to any one of claims 1 to 9 when executed.

12. A computer program product, characterized in that The computer program product comprises program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program instructions, and the processor is used to execute the method according to any one of claims 1 to 9 when executing the program instructions.

14. A vehicle, characterized in that: An electronic device comprising the electronic device described in claim 13.

Images