Vehicle-mounted camera system and vehicle

Abstract

The utility model relates to a kind of vehicle-mounted camera systems and vehicles. Among them, the vehicle-mounted camera system includes: image acquisition component, for continuously photographing to vehicle external environment;Processing device is connected with the image acquisition component, for receiving the frame image of the image acquisition component continuous photographing, output the instruction signal indicating unsafe state avoidance processing, the unsafe state avoidance processing corresponds with the type of the unsafe state of the vehicle-mounted camera system indicated by the frame image;And interface device is connected with the processing device, for receiving and transmitting the instruction signal.According to the vehicle-mounted camera system and vehicle of each aspect of the utility model, the unsafe state of vehicle-mounted camera system can be detected and coped with by using the hardware device of vehicle-mounted camera system itself, and the instruction signal indicating unsafe state avoidance processing is provided, so that unsafe state can be avoided by self-diagnosis and corresponding avoidance processing, and the system reliability and safety are improved.

Description

Technical Field

[0001] This utility model relates to the field of intelligent driving, and in particular to an in-vehicle camera system and a vehicle. Background Technology

[0002] During intelligent driving, vehicles require onboard camera systems to provide clear and accurate images of the surrounding environment in real time. This allows the intelligent driving system to quickly and accurately perceive critical information such as road conditions, traffic signs, pedestrians, and other vehicles. Therefore, the performance of the onboard camera system is crucial to the safety and reliability of intelligent driving.

[0003] To ensure the efficient operation of intelligent driving systems, comprehensive and rigorous anomaly detection of in-vehicle camera systems is essential. When in-vehicle camera systems use components that support ASIL (Automotive Safety Integrity Level), faults in various components, except for some lens and sensor malfunctions, can be detected. However, current detection technologies are not comprehensive or accurate enough. Utility Model Content

[0004] In view of this, the present invention proposes an in-vehicle camera system capable of detecting its own unsafe state and notifying the user of the unsafe state, as well as a vehicle equipped with the in-vehicle camera system.

[0005] According to one aspect of the present invention, a vehicle-mounted camera system is provided, comprising: an image acquisition component for continuously capturing images of the external environment of a vehicle; a processing device connected to the image acquisition component for receiving frame images continuously captured by the image acquisition component and outputting an indication signal indicating unsafe condition avoidance processing, wherein the unsafe condition avoidance processing corresponds to the type of unsafe condition of the vehicle-mounted camera system indicated by the frame images; and an interface device connected to the processing device for receiving and transmitting the indication signal.

[0006] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is any one or a combination of a microcontroller (MCU), an electronic control unit (ECU), a system-on-a-chip (SoC), a neural network processing unit (NPU), and an artificial intelligence image signal processor (AI ISP).

[0007] In one possible implementation of the above-mentioned vehicle-mounted camera system, a bridging integrated circuit (BIC) is also included, wherein the image acquisition component communicates with the processing device via the BIC, and the processing device communicates with the interface device via the BIC.

[0008] In one possible implementation of the above-mentioned vehicle-mounted camera system, the image acquisition component includes: a lens assembly for capturing light signals from the vehicle's external environment; and an image sensor connected to the lens assembly for converting the light signals into electrical signals and generating an image using its own image processing module.

[0009] In one possible implementation of the above-mentioned vehicle-mounted camera system, the interface device is a serializer, i.e., SER, which receives the indication signal and outputs the indication signal through a general purpose input / output pin, i.e., GPIO.

[0010] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is configured to: determine whether there is an unsafe state in the vehicle-mounted camera system based on an anomaly score for the current frame image, an image after self-motion compensation of the vehicle-mounted camera system on the current frame image, and the pattern recognition result of the current frame image using an anomaly detection model.

[0011] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is further configured to: calculate the values ​​of one or more indicators reflecting image quality for the frame image, and generate the abnormal symptom score for the current frame image based on whether the values ​​of the one or more indicators are all below or above a specific threshold for a predetermined duration.

[0012] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is further configured to: estimate the self-motion of the vehicle-mounted camera system based on the current frame image and the previous frame image, and use the estimated self-motion to perform self-motion compensation on the current frame image to obtain the self-motion compensated image.

[0013] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is further configured to: extract the difference between the self-motion compensated image and the image before self-motion compensation as a residual; and perform pattern recognition using the anomaly detection model based on the self-motion compensated image and / or the residual to obtain the pattern recognition result.

[0014] In one possible implementation of the above-described vehicle-mounted camera system, the processing device is further configured to perform the following processing to estimate the self-motion of the vehicle-mounted camera system: estimating coarse and micro motion of the vehicle-mounted camera system based on the current frame image and the previous frame image in the frame image, wherein the coarse motion indicates a motion amplitude greater than the micro motion; calculating the statistical distributions of the coarse and micro motion to determine their respective reliability; and performing a weighted summation of the coarse and micro motion using the reliability for the coarse and micro motion to estimate the self-motion of the vehicle-mounted camera system.

[0015] In one possible implementation of the above-mentioned vehicle-mounted camera system, the processing device is further configured to: comprehensively determine whether there is an unsafe state in the vehicle-mounted camera system by fusing at least two of the abnormal symptom scores, the residual between the self-motion compensated image and the image before self-motion compensation, and the pattern recognition results.

[0016] In one possible implementation of the aforementioned vehicle-mounted camera system, the types of unsafe conditions include: water droplets / dirt adhesion, freezing / blurring, and fog / low illumination; wherein the indication signals include: activating the cleaning system when the type of unsafe condition is water droplets / dirt adhesion; activating the heater when the type of unsafe condition is freezing / blurring; and activating a full AI image processing program when the type of unsafe condition is fog / low illumination, to process the frame image using a full AI model.

[0017] In one possible implementation of the above-mentioned vehicle-mounted camera system, if the processing device still determines that an unsafe state exists after performing the unsafe state avoidance processing, the interface device also notifies the downstream system of the occurrence of the anomaly and the type of the unsafe state.

[0018] According to another aspect of the present invention, a vehicle is provided that includes an in-vehicle camera system according to any of the above implementations.

[0019] By incorporating a processing device within the vehicle-mounted camera system to receive captured frame images and output an indication signal for avoiding unsafe conditions, and transmitting this indication signal via an interface device, the vehicle-mounted camera system and vehicle according to this invention can detect and respond to unsafe conditions using the hardware of the vehicle-mounted camera system itself, and provide an indication signal for avoiding unsafe conditions. Thus, the frame images captured by the vehicle-mounted camera system can be used to indicate how to handle unsafe conditions. Therefore, in cases of image abnormalities caused by lens malfunction, lens obstruction by dirt or water droplets, fog, bad weather, or sensor malfunctions (such as partial power supply abnormalities), the corresponding unsafe conditions can be detected based on the frame images captured by the vehicle-mounted camera system, thereby providing comprehensive and accurate driving decisions.

[0020] Furthermore, the vehicle-mounted camera system can achieve the above capabilities based on its own hardware without relying on downstream systems. Therefore, even if communication with downstream systems is interrupted, it can avoid unsafe conditions through self-diagnosis and corresponding avoidance measures, thereby improving system reliability and stability.

[0021] Other features and aspects of the present invention will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of the present invention together with the specification and serve to explain the principles of the present invention.

[0023] Figure 1 A structural block diagram of an in-vehicle camera system, as an example of prior art, is shown;

[0024] Figure 2 This diagram shows a structural block diagram of a vehicle-mounted camera system according to an embodiment of the present invention;

[0025] Figure 3 This diagram shows a structural block diagram of a specific implementation of an in-vehicle camera system according to an embodiment of the present invention.

[0026] Figure 4 This diagram shows a structural block diagram of another specific implementation of a vehicle-mounted camera system according to an embodiment of the present invention;

[0027] Figure 5 This diagram illustrates a structural block diagram of yet another specific implementation of a vehicle-mounted camera system according to an embodiment of the present invention.

[0028] Figure 6 A schematic flowchart illustrating the processing performed by the processing device of an in-vehicle camera system according to an embodiment of the present invention is shown.

[0029] Figure 7 A schematic flowchart is shown showing another process performed by the processing device of a vehicle-mounted camera system according to an embodiment of the present invention;

[0030] Figure 8 This diagram illustrates the overall process performed by the processing device of a vehicle-mounted camera system according to an embodiment of the present invention.

[0031] Figure 9 A schematic diagram of the overall structure of a camera system according to an embodiment of the present invention is shown. Detailed Implementation

[0032] Various exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0033] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0034] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0035] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of this inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0036] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0037] Furthermore, to better illustrate this utility model, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this utility model can be implemented even without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail, in order to highlight the main points of this utility model.

[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0039] Before providing an exemplary description of the vehicle-mounted camera system according to an embodiment of this utility model, let's first combine... Figure 1 An example of an in-vehicle camera system as an example of prior art is described. This in-vehicle camera system includes: a lens assembly for capturing light signals from the vehicle's external environment; an image sensor for converting the light signals into electrical signals and generating images using its own image processing module; an OSC (Oscillator), connected to the image sensor, serving as an external clock source to provide clock signals to the image sensor; a Ser (Serializer), capable of communicating with the image sensor, for converting parallel image data generated by the image sensor into serial data for transmission to a downstream control module (e.g., an electronic control unit ECU) via a high-speed link (e.g., a coaxial cable); Flash memory, capable of communicating with the image sensor, for storing the image sensor's firmware, calibration data, etc.; a PMIC (Power Management Integrated Circuit), a dedicated power management chip for providing and managing various required voltages for components such as the image sensor; and PoC (Power over Coax), which transmits video data and power simultaneously via a single coaxial cable to simplify wiring and save space.

[0040] This vehicle-mounted camera system uses ASIL-compliant components, enabling it to detect faults in components other than those in the lens assembly and a portion of the image sensor. Specifically:

[0041] OSC: It can detect faults and issue alarms through the clock monitor function of the image sensor.

[0042] Flash: Enables fault detection and alarms via the ROM check function of the image sensor. This is typically performed during startup.

[0043] PoC / PMIC: Enables fault detection and alarms through the power monitoring function of the PMIC.

[0044] Image sensor: It can self-diagnose and issue alarms for faults such as power supply, temperature, CLK, synchronization, software, CPU, hardware (circuit), and control anomalies. Sometimes it cannot detect local power supply anomalies or abnormal images caused by noise.

[0045] Lens assembly: Abnormal condition cannot be detected.

[0046] Ser: It can detect abnormalities in image and communication signals under abnormal conditions and issue alarms.

[0047] Coax / Cable: In the event of an abnormal state of SER-DES (serializer-deserializer), if a chip with ordinary functions (such as those other than R-semi) is used, only a portion of the detection can be performed, such as detecting the presence or absence of a signal or poor signal quality; if an R-semi chip with more advanced functions is used, the abnormality can be identified, including the location of the broken wire.

[0048] In addition, existing functional safety systems primarily detect component malfunctions by monitoring the normality of electronic signals and take appropriate measures (such as issuing alarms) to avoid danger when malfunctions occur. Their workflow includes:

[0049] ADC CCU CDC: Notify drivers

[0050] ADC CCU Communication module: Notifies external systems

[0051] Among them, ADC stands for Automotive Data Concentrator, used to collect information related to vehicle status. CCU stands for Central Control Unit, used to determine anomalies based on information related to vehicle status. CDC stands for Cockpit Display Controller, used to control the instrument panel, central control screen, etc., and can directly interact with the driver.

[0052] Existing vehicle camera systems utilize functional safety systems that detect component failures through electrical testing and notify the driver or external systems of these failures. However, they cannot detect image anomalies caused by lens assembly failures, obstructions from dirt or water droplets, fog, or poor weather conditions, or by sensor malfunctions (such as partial power supply failures). This is because when these anomalies occur, the electrical circuitry is usually functioning correctly; the resulting image is simply blurred, distorted, or obstructed, making it unsuitable for direct analysis of the surrounding environment.

[0053] It is evident that the existing system structure makes it difficult to detect anomalies in vehicle camera systems that cannot be electrically tested.

[0054] To address the aforementioned technical problems, the vehicle-mounted camera system according to this embodiment of the invention includes a processing device that determines whether an unsafe condition exists within the system based on frame images captured by the image acquisition component, and outputs an indication signal to prevent such unsafe conditions from being handled. Therefore, it can identify faults / unsafe conditions that cannot be detected by functional safety systems that rely on electrical signals. Furthermore, since the vehicle-mounted camera system itself can achieve the above capabilities, it can adaptively adjust its state even when communication with external systems is disconnected, thereby providing system reliability and stability.

[0055] Figure 2 A structural block diagram of a vehicle-mounted camera system according to an embodiment of the present invention is shown. Figure 2 As shown, the vehicle-mounted camera system includes:

[0056] Image acquisition component 100 is used to continuously capture images of the vehicle's external environment;

[0057] Processing device 200, connected to image acquisition component 100, is used to receive frame images continuously captured by image acquisition component 100, and output an indication signal indicating unsafe condition avoidance processing, wherein the unsafe condition avoidance processing corresponds to the type of unsafe condition of the vehicle-mounted camera system indicated by the frame images; and

[0058] The interface device 300 is connected to the processing device 200 and is used to receive and transmit the indication signal.

[0059] It should be noted that the "connection" mentioned here can be a physical connection, an electrical connection, or a communicative connection. Figure 2 Arrows are used to illustrate the signal flow between the various modules.

[0060] By incorporating a processing unit within the vehicle-mounted camera system to receive captured frame images and output an indication signal for avoiding unsafe conditions, and transmitting this indication signal via an interface device, the vehicle-mounted camera system of this embodiment can detect and respond to unsafe conditions using its own hardware, and provide an indication signal for avoiding unsafe conditions. Thus, it is possible to indicate how to handle unsafe conditions based on the frame images captured by the vehicle-mounted camera system. Therefore, in cases of image abnormalities caused by lens malfunction, lens obstruction by dirt or water droplets, fog, bad weather, or sensor malfunctions (such as partial power supply abnormalities), the corresponding unsafe conditions can be detected based on the frame images captured by the vehicle-mounted camera system, thereby providing comprehensive and accurate driving decisions.

[0061] Furthermore, the vehicle-mounted camera system according to this embodiment can achieve the above capabilities based on its own hardware without relying on a downstream system. Therefore, even when communication with the downstream system is interrupted, it can avoid unsafe conditions through self-diagnosis and corresponding avoidance processing, thereby improving the system's reliability and stability.

[0062] In one possible implementation, such as Figure 3 As shown, the image acquisition component 100 according to an embodiment of the present invention may include:

[0063] Lens assembly for capturing light signals from the vehicle's external environment; and

[0064] An image sensor, connected to the lens assembly, is used to convert the optical signal into an electrical signal and generate an image using its own image processing module.

[0065] In this implementation, since the processing capabilities of the ISP (Image Processor) within the image sensor are limited, unsafe condition detection in the vehicle camera system can be implemented in the ECU (Electronic Control Unit) SoC (System on Chip) or by adding a new MCU or other processing device. The image acquisition component 100 can form a complete camera product together with the processing device, interface device, and other components; that is, the processing device can be located inside the camera. Alternatively, the processing device can be located outside the camera.

[0066] In one possible specific implementation, such as Figure 3 As shown, the processing device 200 can be a microcontroller (MCU), such as a lightweight microcontroller (MCU). Lightweight MCUs can reduce power consumption while maintaining basic computational accuracy. Therefore, when installed on an in-vehicle camera system, they can reduce the operating load of the camera system.

[0067] It should be noted that, although Figure 3 The MCU has been used as one example of the implementation of the processing device 200, but the present invention is not limited thereto. The processing device 200 according to the embodiments of the present invention can be any one or a combination of a microcontroller (MCU), an electronic control unit (ECU), a system-on-a-chip (SoC), a neural network processing unit (NPU), and an artificial intelligence image signal processor (AI ISP). Reference will be made later. Figure 5 This will explain how the AI ​​ISP is implemented as a processing device 200.

[0068] In one possible specific implementation, such as Figure 4 As shown, the image acquisition component 100 according to an embodiment of the present invention may further include a bridging integrated circuit, i.e., a bridging IC. The image sensor in the image acquisition component 100 communicates with the MCU via the bridging IC, and the MCU communicates with the interface device via the bridging IC. The function of the bridging IC is to provide branching for video or image signals. The interface device 200 can be configured in a manner similar to... Figure 3 Similarly, the general purpose input / output pins (GPIO) of the serializer (SER) are used to output an indication signal indicating that unsafe state avoidance processing should be performed according to the type of unsafe state.

[0069] In one possible specific implementation, such as Figure 5 As shown, the processing device 200 can be an artificial intelligence image processor, i.e., an AI ISP. According to this embodiment, the AI ​​ISP is implemented in a block-defined manner, representing a more intelligent and efficient image processing method. It divides a frame of image into multiple small blocks and analyzes each block for the presence of abnormal features. Then, it summarizes the detection results of each block to determine the overall security status of the image acquisition component. This improves the accuracy of the judgment.

[0070] Furthermore, the AI ​​ISP according to this embodiment can also be based on scene estimation + NR (Noise Reduction). In this implementation, the AI ​​ISP module identifies the current scene based on images captured by the image acquisition component, such as rainy days, tunnels, muddy roads, sunny days with excessively high illumination, and foggy weather with low illumination, and adjusts optimization algorithms such as noise reduction according to the scene to enhance image quality. If the optimized image quality is still below a safety threshold, it is determined that the vehicle-mounted camera system is indeed in an unsafe state. This reduces the false positive rate.

[0071] exist Figure 5The illustrated AI ISP implementation allows for the continuous operation of only the lightweight AI, while a full AI image processing program is activated upon detecting anomalies such as fog or low light. This full AI model processes the frame image. The full AI model, as opposed to the lightweight AI model, refers to the raw AI model without any compression or other processing. The full AI model offers high performance to improve image quality. Therefore, it enables low-power detection of unsafe conditions in automotive camera systems. Integrating this processing device into the automotive camera system meets the stringent requirements of real-time performance and low power consumption in such systems.

[0072] The interface device 200 can be a serializer (SER), which can output indicator signals via general purpose input / output (GPIO) pins. Images processed by the MCU / AI-ISP, etc., are also input to the SER. Under normal circumstances, the output of the image sensor is directly input to the SER; under abnormal circumstances, the image processed by the AI-ISP is input to the SER. The interface device SER can output indicator signals to the downstream system to instruct the downstream system to perform actions corresponding to the indicator signals. Image or video signals are transmitted through the MIPI (Mobile Industry Processor Interface) pin of the interface device 200, while indicator signals for notifying abnormalities and operation commands to start the cleaning system are transmitted via GPIO. The SER can reuse existing SER technologies.

[0073] It should be noted that, Figure 3 , Figure 4 and Figure 5 This is merely an example of a processing device integrated within an in-vehicle camera system. As long as it can output an indication signal representing unsafe condition avoidance handling based on frame images captured by the image acquisition component, the processing device can be installed within the in-vehicle camera system in any manner. The processing device can be implemented by running preset instructions through hardware devices such as MCUs or AI ISPs, or by hardware such as application-specific integrated circuits (ASICs). Although exemplary configurations of the processing device are given below, this application does not limit the specific implementation of the processing device. The processing device can process the frame images using image processing techniques in related technologies to obtain relevant features of the frame images, determine the type of unsafe condition based on the correspondence between the unsafe conditions of the in-vehicle camera system and the features of the frame images, and determine the indication signal based on the correspondence between the type of unsafe condition and the unsafe condition avoidance handling. This application does not limit the specific operation of the processing device.

[0074] compared to Figure 1The prior art shown in the embodiments of the present invention includes a vehicle-mounted camera system with its own processing device, which determines whether an unsafe condition exists in the vehicle-mounted camera system based on frame images captured by the image acquisition component. Therefore, it is possible to identify faults / unsafe conditions that cannot be detected using functional safety systems that rely on electrical signals.

[0075] Furthermore, according to various embodiments of the present invention, when the processing device determines that an unsafe state exists, the vehicle-mounted camera system can also use the interface device to notify the system of unsafe state avoidance processing. Therefore, even when communication with the outside world is disconnected, the system can adaptively adjust its state, thereby providing system reliability and stability.

[0076] In a more specific implementation, the processing device 200 of the vehicle-mounted camera system according to the present invention can be configured to: determine whether there is an unsafe state in the vehicle-mounted camera system based on an anomaly score for the current frame image, an image after self-motion compensation of the vehicle-mounted camera system on the current frame image, and the pattern recognition result of the current frame image using an anomaly detection model.

[0077] By using anomaly symptom scoring, self-motion compensation, and pattern recognition results based on anomaly detection models to determine unsafe conditions, false judgments can be reduced, making the judgment results more stable and reliable. Thus, the vehicle-mounted camera system according to the above embodiments of this utility model can improve its own reliability, thereby ensuring the normal use of the vehicle's intelligent driving functions and enhancing driving safety and convenience.

[0078] Anomaly score can be used to represent the presence of anomalies in an onboard camera system in numerical form (score), and the score can be determined based on the image quality of frame images. In one possible implementation, such as... Figure 6 As shown, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can also be configured to perform step S400. In step S400, for a frame image acquired from the image acquisition component 100, the values ​​of one or more indicators reflecting image quality are calculated, and an abnormality score for the current frame image is generated based on whether the values ​​of the one or more indicators are all below or above a specific threshold for a predetermined duration.

[0079] One or more of these indicators can be, for example, image brightness, contrast, sharpness, and edge sharpness (image edge gradient). Multiple indicators can be compared individually to their respective thresholds, and combined to identify abnormal signs. For example, the overall contrast (90% / 10% ratio of the brightness histogram) and sharpness (image edge gradient) of frames captured by an in-vehicle camera system can be continuously monitored. When both are below a specific threshold for a prolonged period, it is determined that there may be abnormal signs such as lens fogging / condensation. Similarly, the overall brightness value of frames captured by an in-vehicle camera system can be continuously monitored. When the overall brightness value is consistently significantly higher than a specific threshold for a prolonged period, it can be considered an abnormal sign such as overexposure caused by direct sunlight.

[0080] According to the processing device 200 of this embodiment, after calculating the values ​​of one or more indicators reflecting image quality, it compares them with an anomaly threshold. However, instead of screening for specific anomalies, it continuously monitors for a period of time and judges anomalies based on whether the indicator value is consistently below or above the anomaly threshold during the duration. For example, only when the contrast ratio remains below a predetermined threshold for a continuous period is it determined that there may be anomalies such as fog, water droplets, or icing. In this way, by continuously statistically analyzing the values ​​of one or more indicators reflecting image quality and judging the existence of anomalies based on the indicator values ​​over a continuous period, the processing device 200 of this embodiment can ensure the accuracy of anomaly judgment while reducing the false judgment rate. Therefore, it can provide a more accurate basis for judging unsafe conditions.

[0081] Furthermore, for frame images determined to have abnormal symptoms, an abnormality score of, for example, 1 can be generated; for frame images determined not to have abnormal symptoms, an abnormality score of, for example, 0 can be generated. This allows for the differentiation between frame images with and without abnormal symptoms. Of course, the abnormality score is not limited to 1 and 0; those skilled in the art can set it according to specific application scenarios, as long as it can distinguish between frame images with and without abnormal symptoms.

[0082] The self-motion of an in-vehicle camera system is a quantitative representation of the system's own motion. It can be represented by the distance the system moves within a preset coordinate system between the capture of two adjacent frames. In one possible implementation, such as... Figure 6As shown, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can also be configured to perform step S500. In step S500, based on the current frame image and the previous frame image, the self-motion of the vehicle-mounted camera system is estimated, and the estimated self-motion is used to perform self-motion compensation on the current frame image to obtain the self-motion compensated image. Through self-motion compensation, the reliability of the judgment result of the unsafe state of the step can be further improved.

[0083] In one possible specific implementation, such as Figure 7 As shown, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can also be configured to perform the processing steps S510 to S530 to estimate the self-motion of the vehicle-mounted camera system.

[0084] In step S510, based on the current frame image and the previous frame image in the frame image, the coarse movement and micro movement of the vehicle-mounted camera system are estimated, wherein the coarse movement indicates a greater range of motion than the micro movement.

[0085] Coarse motion reflects the self-motion of the onboard camera system caused by large vehicle movements such as turning and changing lanes. Micro motion reflects smaller motion amplitudes, such as those caused by smooth or subtle movements of the vehicle. Based on this step, it is possible to effectively estimate larger motion amplitudes caused by vehicle turning and changing lanes, as well as subtle and smooth movements that cannot be captured by coarse motion.

[0086] When estimating coarse movement, the resolution of the frame image can be reduced first to obtain a low-resolution frame image. This can significantly reduce the computational load, thereby improving computational efficiency and meeting real-time requirements.

[0087] In one possible specific implementation, the coarse movement (dx1, dy1) of the vehicle-mounted camera system can be estimated by calculating the correlation between the current frame image and the previous frame image in the low-resolution frame image.

[0088] The following is an example of how to calculate coarse motion. Those skilled in the art should understand that other methods can also be used to calculate larger offsets between images as coarse motion.

[0089] Perform a Fourier transform on the images of the current frame and the previous frame, and calculate the displacement using the phase difference.

[0090] Use the position with the highest peak as the offset.

[0091] The sharper the peak, the higher the confidence level should be.

[0092] In one possible specific implementation, the coarse movement (dx1, dy1) of the vehicle-mounted camera system can be obtained through the following steps:

[0093] (1) Perform Fourier transform on the previous frame image and the current frame image using Formula 1, where the brightness distribution of the image is decomposed into amplitude and phase.

[0094] , Formula 1

[0095] in, Let f(x,y) represent the previous frame image, F(u,v) represent the result of the Fourier transform of the previous frame image, g(x,y) represent the current frame image, and G(u,v) represent the result of the Fourier transform of the current frame image.

[0096] (2) Calculate the normalized cross spectrum using Formula 2, where the Fourier transform results of the two images are multiplied and normalized, and only the phase difference is extracted.

[0097] Formula 2

[0098] (3) Use Formula 3 to perform inverse Fourier transform to restore the original image coordinate system.

[0099] Formula 3

[0100] in, This represents the inverse Fourier transform.

[0101] (4) Detect and read the peak position, and set the position of the highest peak as the offset as in Formula 4.

[0102] Formula 4

[0103] In Formula 4, (Δx, Δy) represents the coarse movement (dx1, dy1) of the vehicle-mounted camera system.

[0104] Additionally, peak height and sharpness can be further measured to calculate reliability. For example, as shown in Formula 5, the PSR (Peak to Sidelobe Ratio) can be used for reliability assessment, or it can be divided by 10 to obtain a normalized numerical reliability W1. This reliability W1 will be used in step S550, mentioned later, to estimate the self-motion of the vehicle-mounted camera system. The reliability can also be determined based on other statistical distributions related to the coarse motion calculation, as long as they reflect the accuracy of the obtained coarse motion.

[0105] Formula 5

[0106] Where p represents the peak height, This represents the average value around the peak (the annular region). This represents the standard deviation around the peak (the annular region).

[0107] It should be noted that the above is only one possible example of calculating the coarse movement of the vehicle-mounted camera system. The processing performed by the processing device 200 according to the embodiment of this utility model is not limited to this. Those skilled in the art can use other processing methods to calculate the coarse movement of the vehicle-mounted camera system. As long as the coarse movement of the vehicle-mounted camera system is used to estimate its self-motion when performing unsafe condition detection of the vehicle-mounted camera system, it falls within the protection scope of this utility model.

[0108] In step S510, a lightweight AI model can be used to generate feature maps of the low-resolution frame image, and the correlation between the feature maps of the current frame image and the feature maps of previous frames in the low-resolution frame image can be calculated to estimate the micro-motion (dx2, dy2) of the vehicle-mounted camera system. The correlation between feature maps can be calculated based on optical flow estimation. Therefore, the micro-motion can be calculated from the optical flow estimation of the feature maps, for example, by first generating feature maps using an AI model and then performing optical flow estimation based on the feature maps. A lightweight AI model can refer to an artificial intelligence model that significantly reduces the number of parameters, computational complexity, and resource consumption while maintaining core performance through optimized algorithm design or compression techniques. This reduces power consumption, and this step can be performed on the camera side.

[0109] In one possible implementation, the micro-motion can be obtained through the following approach. Those skilled in the art will understand that even finer, more precise offsets of the image can be calculated as micro-motion using other methods.

[0110] By analyzing the gradient and temporal changes of an image (such as the feature map mentioned above), we can calculate how much a point has moved in which direction.

[0111] The image (such as the feature map mentioned above) is segmented into blocks, and motion vectors are inferred by optical flow estimation for each block.

[0112] The outlier exclusion process described later is implemented based on the motion vectors calculated here.

[0113] The background vector is calculated using the Inlier Ratio (the proportion of vectors in the whole that are consistent with the average motion), and the reliability W2 is calculated.

[0114] More specifically, Equation 6 can be used to estimate the micro-movement (dx2, dy2) of the vehicle-mounted camera system:

[0115] Formula 6

[0116] Where I represents image intensity, x and y represent pixel positions, t represents time (frame number), I / x represents the horizontal gradient of the image, I / y represents the vertical gradient of the image, I / t represents the brightness change between frames, and (dx, dy) is the micro-movement of the vehicle camera system (dx2, dy2).

[0117] Furthermore, if the background vector is regularized, the reliability W2 is high. For example, the Inlier Ratio can be used as an indicator of reliability W2. The Inlier Ratio can be calculated as follows: calculate the difference between each block vector and the average vector; the number of differences less than a threshold is denoted as N inlier, and the total number of block vectors is denoted as Ntotal; then, Inlier ratio = N inlier / Ntotal. The reliability W2 mentioned here will be used in step S550 mentioned later when estimating the self-motion of the vehicle-mounted camera system. The reliability can also be determined based on other statistical distributions related to the calculation of micro-motion, as long as they can reflect whether the obtained micro-motion is accurate.

[0118] It should be noted that the above are only a few possible examples of implementations for calculating the coarse and micro movements of the vehicle-mounted camera system. The processing performed by the processing device 200 of the vehicle-mounted camera system according to the embodiments of this utility model is not limited to these. Those skilled in the art can use other processing methods to calculate the coarse and micro movements of the vehicle-mounted camera system. As long as the coarse and micro movements of the vehicle-mounted camera system are used to estimate its self-motion when performing unsafe condition detection of the vehicle-mounted camera system, it falls within the protection scope of this utility model.

[0119] In step S520, the statistical distributions of the coarse and micro shifts are calculated to determine their respective reliability W1 and W2. For example, steps S510 and S520 can be performed on each frame image to obtain the coarse and micro shifts when each frame image is used as the current frame image. The statistical distributions of these coarse and micro shifts are then calculated, and the reliability can be determined using the consistency of the distribution and background integration (sharpness of the correlation peak) as indicators. For the specific implementation method of determining reliability W1 and W2, please refer to the description of the estimation processing of coarse and micro shifts above, which will not be repeated here.

[0120] Additionally, it can identify regions with significantly different motion from their surroundings (outlier regions), classifying these regions as moving objects and removing their corresponding vector data. For example, the image can be segmented into blocks, and the correlation coefficient vector of each block can be calculated. By analyzing the vector distribution, regions with vectors that deviate significantly from their surroundings can be identified as moving objects and excluded from self-motion compensation. The presence of moving objects can cause background compensation to be shifted, potentially increasing the overall residual. After self-motion compensation, the portion outside the background will appear as a residual. Within the residual, portions that maintain the same position and shape for an extended period will be detected as stains or water droplets. Therefore, residuals can be used for stain or water droplet detection.

[0121] In one possible implementation, scene classification using a lightweight AI model can also be performed simultaneously, such as daytime, nighttime, backlighting, rain, fog, and ultra-low light. The anomaly detection threshold and reliability values ​​W1 and W2 used in generating anomaly symptom scores can be automatically adjusted based on the scene type. For example, at night, since contrast is inherently reduced, the anomaly detection threshold can be temporarily lowered. Additionally, at night, there are more moving objects, so the reliability W1 for coarse movement can be appropriately increased.

[0122] In step S530, the coarse movement and the micro movement are weighted and summed using the reliability W1 for the coarse movement and the reliability W2 for the micro movement to estimate the self-motion quantity (dx, dy) of the vehicle-mounted camera system. A higher reliability W1 is achieved by emphasizing the coarse movement, while a higher reliability W2 is achieved by emphasizing the micro movement. For example, the self-motion quantity of the vehicle-mounted camera system can be estimated using Equation 7:

[0123] (dx, dy) = W1 × (dx1, dy1) + W2 × (dx2, dy2) Formula 7,

[0124] Where W1 + W2 = 1.

[0125] In this way, the two-stage self-motion compensation based on the (phase) correlation of low-resolution images and the (phase) correlation of AI feature maps suppresses the false correlation caused by self-motion / moving bodies, thus stabilizing the detection.

[0126] Furthermore, during step S500 in the processing device 200, motion compensation is performed on the current frame image using the estimated self-motion amount to obtain the motion-compensated image. In one possible implementation, the current frame can be shifted in the opposite direction based on the estimated self-motion amount of the camera in the previous frame. If the estimated camera movement is dx to the right and dy downwards, the current frame is shifted dx to the left and dy upwards.

[0127] As described above, by removing the moving object and estimating the self-motion using only the background, the non-background portion will form a residual after self-motion compensation. In this residual, portions that maintain the same position and shape for a long time will be detected as stains. Therefore, this residual can be used for stain detection.

[0128] In one possible implementation, such as Figure 6 As shown, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can also be configured to perform step S600. In step S600, the difference between the image after motion compensation and the image before motion compensation is extracted as a residual, and pattern recognition is performed using the anomaly detection model based on the image after motion compensation and / or the residual to obtain the pattern recognition result.

[0129] In abnormal situations, such as when foreign objects are attached to the lens, these objects will not follow the lens's own motion model. Therefore, as mentioned above, these areas will stand out in the residuals. Inputting the residuals into an anomaly detection model allows us to determine the type of anomaly in that area. The anomaly detection model can be a lightweight AI model, such as a trained convolutional neural network. By inputting motion-compensated frame images and / or residuals, it can output matching results against predefined anomaly patterns. Anomaly patterns could be, for example, raindrop / water stain patterns, mud / dirt patterns, snow / ice cover patterns, etc. The matching results can include pattern labels and corresponding confidence scores, which can be used to subsequently determine the type of unsafe condition by combining anomaly symptom scoring.

[0130] In one possible implementation, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention may also be configured to: comprehensively determine whether there is an unsafe state in the vehicle-mounted camera system by weighted fusion of at least two of the abnormal symptom scores, the residuals, and the pattern recognition results.

[0131] In one possible implementation of this method, for example, if in step S600, the anomaly detection model identifies the presence of "raindrops" with high confidence for the current frame image, and a corresponding speckled pattern also appears in the residual of the current frame image, and in step S400, an anomaly score is generated due to the image clarity index of the current frame image being consistently low over a long period of time, then the processing device 200 can determine with great certainty that the unsafe condition of "raindrop obstruction" exists in the vehicle camera system.

[0132] In another possible implementation, if there is a slight residual in the residual of the current frame image, but the anomaly detection model does not identify any known anomaly pattern for the current frame image, and generates an anomaly score for the current frame image that does not have any anomaly signs, then the system may consider the residual as noise and determine that there is no unsafe condition in the vehicle system.

[0133] Furthermore, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can also select some of the above three items as the basis for judging unsafe conditions based on environmental factors such as different time periods and different weather conditions. For example, during the daytime, dynamic recognition is highly effective and AI is stable, so the focus is on the combination of residual and AI (i.e., anomaly detection model); at night, because the reliability of residual is reduced, the focus is on the combination of image statistics (i.e., anomaly sign scoring) and AI; and in rainy weather, the combination of residual + AI is preferred.

[0134] In another possible implementation, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention can further convert the abnormal sign score, the residual between the image after self-motion compensation and the image before self-motion compensation, and the pattern recognition result into corresponding numerical values, weight the numerical values, perform calculations based on the weighted numerical values, and determine whether an unsafe state exists based on the calculation results. For example, the abnormal sign score can be represented by 0 or 1. If the residual shows a part that maintains the same position and shape for a long time, the value corresponding to the residual is 1; otherwise, it is 0. Each type of detection result corresponding to the anomaly detection model can be assigned a corresponding numerical value. For example, 1, 2, and 3 represent three types, and 0 represents no type. The weights corresponding to the numerical values ​​can be set as needed. For example, as mentioned above, the weights can be set according to time period or environmental factors, or the confidence level of the classification result of the anomaly detection model for each category can be used as the weight of the corresponding numerical value of the classification result. The weighted values ​​can be obtained based on a preset calculation method. This application does not restrict the specific calculation method. For example, the value corresponding to the residual and the value of the classification result of water droplets / dirt output by the anomaly detection model can be weighted and summed as the value corresponding to the water droplet / dirt classification result. The weighted anomaly score can be used as an accumulation term or a multiplication term to control the final classification result.

[0135] Thus, the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention combines all or part of the following three factors to determine whether there is an unsafe state in the vehicle-mounted camera system: the abnormal sign score for the current frame image, the residual between the current frame image after self-motion compensation by the vehicle-mounted camera system and the image before self-motion compensation, and the pattern recognition result of the current frame image using an anomaly detection model. This allows for the determination of whether there is an unsafe state in the vehicle-mounted camera system, thereby ensuring a high detection rate while minimizing the false alarm rate.

[0136] The following describes the further beneficial effects of the vehicle-mounted camera system according to embodiments of the present invention, in conjunction with existing technology.

[0137] Patent document 1 (CN118247694A) discloses a method for detecting occlusion based on time-series changes in statistical computing block units. The structure of this patent document is as follows:

[0138] ① Divide each frame of the video captured by the camera into multiple image blocks of the same size.

[0139] ② Determine the group of blocks that correspond in spatial position between adjacent frames.

[0140] ③ Calculate the image similarity between corresponding blocks. Specifically, use features such as pixel gradient intensity and angle to quantify the visual consistency between blocks.

[0141] ④ Calculate the rate of change of similarity of each block within a specific time period.

[0142] ⑤ When the rate of change in similarity is low (the appearance of the block hardly changes even if the frame changes), it is determined that the block comes from an obstruction on the lens.

[0143] This method triggers occlusion detection based on minimal image changes between frames. However, when the background image moves significantly due to a vehicle turning or accelerating / decelerating while part of the lens is obscured, the blocks captured from completely different subjects are compared, inevitably reducing the similarity between them. This leads to a decrease in the inferred similarity of the occluded area, making detection impossible. Therefore, Patent Document 1 has the following problems:

[0144] Question 1: The detection sensitivity decreases due to local occlusion caused by self-motion.

[0145] Patent documents 2 (CN112804522B), 3 (US2025046075A1), and 4 (CN112446246B) all involve technologies that use AI models to detect anomalies (dirt, obstructions, etc.) in images from vehicle-mounted cameras. These technologies are characterized by inputting the acquired images into an AI model to determine the presence and type of anomalies such as water, mud, dirt, and overexposure.

[0146] Patent Document 2 is specifically designed for detecting unsafe conditions in stationary states. However, background changes in the image caused by vehicle movement (acceleration, turning) are misidentified as lens dirt or occlusion, leading to a decrease in detection accuracy after the vehicle begins to move. Patent Document 3 uses the unreliability (reliability) estimation results of the AI ​​model itself to reduce the risk of false detection / misjudgment to some extent. However, because it only uses the introspective unreliability estimation results of a single AI model, the risk of misjudgment in unlearned scenarios remains high. Therefore, Patent Documents 2 and 3 have the following problems:

[0147] Question 2: Risk of false detection / false judgment.

[0148] In Patent Document 4, since the entire image is processed pixel by pixel, the requirements for accuracy and detail are very high. However, this high-precision detection requires a large-scale AI model, and if it is always performed in a resource-constrained environment such as an in-vehicle camera / ECU, it will lead to excessive processing load and heat generation. Therefore, Patent Document 4 has the following problems:

[0149] Question 3: Handling the trade-off between load and system.

[0150] According to the vehicle-mounted camera system of the above embodiments of this utility model, for the images acquired by its image acquisition component, the self-motion of the camera is compensated based on two phase correlation relationships: coarse correlation and fine correlation. This can remove image changes in appearance caused by changes in camera posture or vibration. In other words, this utility model suppresses erroneous correlations caused by self-motion / moving bodies through two-stage self-motion compensation, thereby stabilizing the detection and solving the aforementioned problem 1.

[0151] Furthermore, the vehicle-mounted camera system according to the above embodiments of this utility model uses image statistical information and lightweight AI-based pattern matching and self-motion compensation of low-resolution images to determine unsafe conditions. The self-motion compensation is characterized by calculation based on the phase correlation of the low-resolution image and the phase correlation of the AI ​​feature map, thus further increasing reliability. By integrating the two, anomalies can be detected stably while reducing false detections. Therefore, this utility model achieves robustness and suppresses false detections through a combination of statistics (anomaly symptom extraction) + AI (feature map generation / pattern matching) + self-motion compensation, thereby solving the aforementioned problem 2.

[0152] Furthermore, according to the vehicle-mounted camera system of the above embodiments of this utility model, only the lightweight AI is always running (the high-precision AI is only activated when an anomaly is detected, as will be described later), thus achieving low-power detection of dirt on the camera lens. water droplets This eliminates unsafe conditions such as blurriness, thus solving problem 3 mentioned above. Therefore, the vehicle-mounted camera system according to the above embodiments of this utility model can operate from the camera side.

[0153] In addition to solving the aforementioned problems 1-3 in the prior art, the vehicle-mounted camera system according to the above embodiments of the present invention also produces the following technical effects compared with the prior art:

[0154] Effect 1: Even at night, in backlight, or in the rain, self-motion compensation is unlikely to fail;

[0155] Effect 2: It can reliably separate dirt and moving objects based on their persistence over time and spatial characteristics;

[0156] Effect 3: Due to its low power consumption, the process can be completed at the camera side, thus eliminating reliance on the ECU and reducing system costs.

[0157] In one possible implementation, the types of unsafe states mentioned in the above embodiments may include: water droplet / dirt adhesion, freezing / blurring, fog / low light. The indication signal transmitted by the interface device 300 may include:

[0158] If the unsafe condition is caused by water droplets / dirt adhesion, the cleaning system will be activated.

[0159] If the type of unsafe condition is frozen / fuzzy, activate the heater;

[0160] When the unsafe condition is fog / low light, a full AI image processing program is initiated to process the frame image using a full AI model to improve accuracy. Here, the full AI model, as opposed to a lightweight AI model, refers to the original AI model that has not undergone compression or other processing. The full AI model can provide high performance to improve image quality.

[0161] In one possible specific implementation, for example, the type of unsafe state and the corresponding indication signal for unsafe state avoidance processing can be output as shown in Table 1.

[0162] Table 1: Abnormal Avoidance Measures and Notification Content by Category

[0163]

[0164] In one possible implementation, for the vehicle-mounted camera system according to the above embodiments of the present invention, if the system fails to return to normal after performing the unsafe state avoidance process, the interface device 300 also notifies the downstream system of the occurrence of the anomaly and the type of the unsafe state, in order to eliminate the unsafe state as soon as possible.

[0165] To facilitate a better understanding of the technical content of this utility model Figure 8 A flowchart illustrating the overall processing performed by the processing device 200 of the vehicle-mounted camera system according to an embodiment of the present invention is shown. Specifically, step ① can be found in the description related to step S400 performed by the processing device 200; step ② can be found in the description related to step S510 performed by the processing device 200; step ③ can be found in the description related to step S510 performed by the processing device 200; step ④ can be found in the description related to step S510 performed by the processing device 200; steps ⑤⑥⑦⑧ can be found in the description related to steps S520 and S530 performed by the processing device 200; step ⑨ can be found in the description related to step S600 performed by the processing device 200; and step ⑩ can be found in the description related to the unsafe condition judgment performed by the processing device 200. Please refer to the description related to interface device 300. It will not be repeated here.

[0166] Thus, the vehicle-mounted camera system according to the above embodiments of this utility model combines image statistical information with pattern matching and self-motion compensation based on lightweight AI for low-resolution images to determine unsafe conditions. The self-motion of the camera is compensated based on two phase correlation relationships: coarse correlation and micro correlation. Therefore, it can perform high-precision detection with low false recognition rate in real time with low power consumption. This allows for timely and accurate detection of unsafe conditions in the vehicle-mounted camera system and enables targeted responses, thereby improving the reliability of the vehicle-mounted camera system.

[0167] Figure 9 This diagram illustrates the overall structure of a camera system according to an embodiment of the present invention. Component A may correspond to the image acquisition assembly 100 described above, and component B may correspond to the processing device 200 described above. Possible implementations are described above and will not be repeated here.

[0168] This utility model also relates to a vehicle, in one possible implementation of which the vehicle may include the vehicle-mounted camera system according to any of the above embodiments of this utility model.

[0169] According to an embodiment of the present invention, the vehicle's onboard camera system is equipped with a processing device that can determine whether an unsafe condition exists in the onboard camera system based on frame images captured by the image acquisition component, and output an indication signal indicating that the unsafe condition should be avoided. Therefore, it is possible to identify faults / unsafe conditions that cannot be detected by functional safety systems that rely on electrical signals. Furthermore, since the onboard camera system itself can achieve the above capabilities, it can adaptively adjust its state even when communication with the outside world is disconnected, thereby improving vehicle driving safety.

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0171] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A vehicle-mounted camera system, characterized in that, include: Image acquisition component for continuous imaging of the vehicle's external environment; A processing device, connected to the image acquisition component, is used to receive frame images continuously captured by the image acquisition component and output an indication signal indicating unsafe condition avoidance processing, wherein the unsafe condition avoidance processing corresponds to the type of unsafe condition of the vehicle-mounted camera system indicated by the frame image; as well as An interface device, connected to the processing device, is used to receive and transmit the indication signal.

2. The vehicle-mounted camera system according to claim 1, characterized in that, The processing device is any one or a combination of a microcontroller (MCU), an electronic control unit (ECU), a system-on-a-chip (SoC), a neural network processing unit (NPU), and an artificial intelligence image signal processor (AI ISP).

3. The vehicle-mounted camera system according to claim 2, characterized in that, It also includes a bridging integrated circuit, i.e., a bridging IC, wherein the image acquisition component communicates with the processing device via the bridging IC, and the processing device communicates with the interface device via the bridging IC.

4. The vehicle-mounted camera system according to any one of claims 1 to 3, characterized in that, The image acquisition component includes: Lens assembly for capturing light signals from the vehicle's external environment; and An image sensor, connected to the lens assembly, is used to convert the optical signal into an electrical signal and generate an image using its own image processing module.

5. The vehicle-mounted camera system according to any one of claims 1 to 3, characterized in that, The interface device is a serializer, i.e., SER. The SER receives the indication signal and outputs the indication signal through a general purpose input / output pin, i.e., GPIO.

6. The vehicle-mounted camera system according to claim 1, characterized in that, The types of unsafe conditions include: water droplets / dirt adhesion, freezing / blurring, fog / low light; The indication signal includes: If the unsafe condition is caused by water droplets / dirt adhesion, the cleaning system will be activated. If the type of unsafe condition is frozen / fuzzy, activate the heater; When the unsafe condition is of the type of fog / low light, a full AI image processing program is initiated to process the frame image using a full AI model.

7. The vehicle-mounted camera system according to claim 1, characterized in that, If the processing device still determines that an unsafe state exists after performing the unsafe state avoidance process, the interface device also notifies the downstream system of the occurrence of the anomaly and the type of the unsafe state.

8. A vehicle, characterized in that, Including the vehicle-mounted camera system according to any one of claims 1 to 7.

Images