Method and device for processing unsafe state in vehicle-mounted camera system, storage medium and program product

Abstract

The invention relates to a method and a device for processing an unsafe state in a vehicle-mounted camera system, a storage medium and a program product. The processing method comprises the following steps: acquiring frame images continuously shot by a vehicle-mounted camera system; judging whether an unsafe state exists in the vehicle-mounted camera system or not according to an abnormal symptom score for the current frame image, an image obtained after self-motion compensation of the vehicle-mounted camera system is carried out on the current frame image, and a mode recognition result carried out on the current frame image by using an abnormal detection model; and under the condition that the unsafe state exists, executing unsafe state avoiding processing according to the type of the unsafe state. According to the method and the device for processing the unsafe state in the vehicle-mounted camera system, the storage medium and the program product, the unsafe state existing in the vehicle-mounted camera system can be timely and accurately found, and targeted response can be performed, so that the reliability of the vehicle-mounted camera system is improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving, and more particularly to a method and apparatus for handling unsafe states in an in-vehicle camera system, as well as a storage medium and program product. Background Technology

[0002] During intelligent driving, vehicles rely on onboard camera systems to provide images of the surrounding environment. If the camera lenses are damaged by dirt, water droplets, or freezing, the images provided will not accurately inform driving decisions, potentially leading to unsafe incidents. Therefore, it is necessary to address unsafe conditions in onboard camera systems. Summary of the Invention

[0003] In view of this, this disclosure provides a method and apparatus for handling unsafe conditions in a vehicle-mounted camera system, as well as a storage medium and program product.

[0004] According to one aspect of this disclosure, a method for handling unsafe conditions in an in-vehicle camera system is provided, comprising: acquiring frame images continuously captured by the in-vehicle camera system; determining whether an unsafe condition exists in the in-vehicle camera system based on an anomaly symptom score for the current frame image, an image after self-motion compensation of the current frame image by the in-vehicle camera system, and a pattern recognition result performed on the current frame image using an anomaly detection model; and, if the unsafe condition exists, performing unsafe condition avoidance processing according to the type of the unsafe condition.

[0005] In one possible implementation of the above processing method, it further includes: calculating the values ​​of one or more indicators reflecting image quality for the frame image, and generating the abnormal symptom score for the current frame image based on whether the values ​​of the one or more indicators are all below or above a specific threshold within a predetermined duration.

[0006] In one possible implementation of the above processing method, it further includes: estimating the self-motion of the vehicle-mounted camera system based on the current frame image and the previous frame image, and using the estimated self-motion to perform self-motion compensation on the current frame image to obtain the self-motion compensated image.

[0007] In one possible implementation of the above processing method, it further includes: extracting the difference between the self-motion compensated image and the image before self-motion compensation as a residual; and using the anomaly detection model to perform pattern recognition based on the self-motion compensated image and / or the residual to obtain the pattern recognition result.

[0008] In one possible implementation of the above processing method, estimating the self-motion of the vehicle-mounted camera system includes: estimating the coarse motion and micro motion of the vehicle-mounted camera system based on the current frame image and the previous frame image in the frame image, wherein the coarse motion indicates a motion amplitude greater than the micro motion; calculating the statistical distribution of the coarse motion and the micro motion to determine their respective reliability; and performing a weighted summation of the coarse motion and the micro motion using the reliability for the coarse motion and the reliability for the micro motion to estimate the self-motion of the vehicle-mounted camera system.

[0009] In one possible implementation of the above processing method, determining whether an unsafe state exists in the vehicle-mounted camera system includes: comprehensively determining whether an unsafe state exists in the vehicle-mounted camera system by fusing at least two of the abnormal symptom scores, the residual between the image after self-motion compensation and the image before self-motion compensation, and the pattern recognition results.

[0010] In one possible implementation of the above processing method, the types of unsafe conditions include: water droplets / dirt adhesion, freezing / blurring, and fog / low illumination; wherein, performing unsafe condition avoidance processing includes: activating the cleaning system when the type of unsafe condition is water droplets / dirt adhesion; activating the heater when the type of unsafe condition is freezing / blurring; and activating a full AI image processing program to process the frame image using a full AI model when the type of unsafe condition is fog / low illumination.

[0011] In one possible implementation of the above processing method, it further includes: if the normal state cannot be restored after performing the unsafe state avoidance processing, notifying the downstream system of the occurrence of the anomaly and the type of the unsafe state.

[0012] According to another aspect of this disclosure, an apparatus for handling unsafe states in a vehicle-mounted camera system is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0013] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0014] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0015] By determining whether an unsafe condition exists in the vehicle camera system based on the anomaly score of the current frame image captured by the vehicle camera system, the image after self-motion compensation of the current frame image by the vehicle camera system, and the pattern recognition result of the current frame image using an anomaly detection model, and performing unsafe condition avoidance processing according to the type of unsafe condition, the unsafe condition handling method and apparatus, storage medium and program product of this disclosure can promptly and accurately detect unsafe conditions in the vehicle camera system and take targeted measures, thereby improving the reliability of the vehicle camera system.

[0016] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0018] Figure 1 A flowchart illustrating a method for handling unsafe conditions in an in-vehicle camera system according to an embodiment of the present disclosure is shown.

[0019] Figure 2 A flowchart illustrating a method for handling unsafe conditions in a vehicle-mounted camera system according to another embodiment of the present disclosure;

[0020] Figure 3 A flowchart is shown illustrating the process of estimating the self-motion of an in-vehicle camera system and performing self-motion compensation processing in a processing method according to another embodiment of the present disclosure;

[0021] Figure 4 A flowchart illustrating an overall method for handling unsafe conditions in an in-vehicle camera system according to an embodiment of the present disclosure is provided.

[0022] Figure 5 A structural block diagram of an apparatus for handling unsafe conditions in a vehicle-mounted camera system according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0024] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0025] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0026] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0027] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0028] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0029] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0030] Figure 1 A flowchart illustrating a method for handling unsafe conditions in an in-vehicle camera system according to an embodiment of the present disclosure is provided. This method can be executed by an in-vehicle terminal (or in-vehicle computing device) or by a server. For example, frame images captured by the in-vehicle camera system can be uploaded to a server, and the server processes them based on the method described in the embodiment of the present disclosure, then returns the processing result to the in-vehicle terminal. Figure 1 As shown, the processing method includes:

[0031] Step S100: Acquire frame images continuously captured by the vehicle-mounted camera system;

[0032] Step S200: Based on the abnormality score for the current frame image, the image after self-motion compensation by the vehicle-mounted camera system for the current frame image, and the pattern recognition result of the current frame image using the anomaly detection model, determine whether there is an unsafe condition in the vehicle-mounted camera system; and

[0033] Step S300: If the unsafe condition exists, perform unsafe condition avoidance processing according to the type of the unsafe condition. If the unsafe condition does not exist, the continuously captured frame images from the vehicle-mounted camera system are output normally to the subsequent stage.

[0034] In one possible implementation, the vehicle-mounted camera system can be a camera. When powered on, the camera can continuously acquire images of the vehicle's surrounding environment. This enables real-time, uninterrupted monitoring of the surrounding environment, ensuring that the system can promptly capture any sudden, transient unsafe events, thereby quickly executing unsafe condition avoidance measures and improving the reliability of the vehicle-mounted camera system.

[0035] In addition, as in step S200, this disclosure uses anomaly symptom scoring, self-motion compensation, and pattern recognition results based on anomaly detection models to determine unsafe states, which can reduce false judgments and make the judgment results more stable and reliable.

[0036] Thus, through the above steps S100 to S300, the processing method according to the above embodiments of this disclosure can improve the reliability of the vehicle camera system, thereby ensuring the normal use of the vehicle's intelligent driving function and improving driving safety and convenience.

[0037] Anomaly score can be used to represent the presence of anomalies in an onboard camera system in numerical form (score), and the score can be determined based on the image quality of frame images. In one possible implementation, such as... Figure 2 As shown, the method for handling unsafe conditions in a vehicle-mounted camera system according to another embodiment of this disclosure may further include step S400. In step S400, for the frame image acquired in step S100, the values ​​of one or more indicators reflecting image quality are calculated, and the abnormal symptom score for the current frame image is generated based on whether the values ​​of the one or more indicators are all below or above a specific threshold for a predetermined duration.

[0038] One or more of these indicators can be, for example, image brightness, contrast, sharpness, and edge sharpness (image edge gradient). Multiple indicators can be compared individually to their respective thresholds, and combined to identify abnormal signs. For example, the overall contrast (90% / 10% ratio of the brightness histogram) and sharpness (image edge gradient) of frames captured by an in-vehicle camera system can be continuously monitored. When both are below a specific threshold for a prolonged period, it is determined that there may be abnormal signs such as lens fogging / condensation. Similarly, the overall brightness value of frames captured by an in-vehicle camera system can be continuously monitored. When the overall brightness value is consistently significantly higher than a specific threshold for a prolonged period, it can be considered an abnormal sign such as overexposure caused by direct sunlight.

[0039] According to the processing method of this disclosure, after calculating the values ​​of one or more indicators reflecting image quality, they are compared with an anomaly threshold. However, at this time, specific anomaly screening is not performed; instead, monitoring is continuously conducted for a period of time, and anomaly signs are judged based on whether the indicator value is consistently lower than or higher than the anomaly threshold during the duration. For example, anomalies such as fog, water droplets, or icing are only determined when the contrast ratio remains consistently lower than a predetermined threshold during the duration. In this way, by continuously statistically analyzing the values ​​of one or more indicators reflecting image quality and judging the existence of anomalies based on the indicator values ​​over a period of time, the processing method of this disclosure can ensure the accuracy of anomaly judgment while reducing the false judgment rate. Therefore, it can provide a more accurate basis for the unsafe state judgment in step S200.

[0040] Furthermore, for frame images determined to have abnormal symptoms, an abnormality score of, for example, 1 can be generated; for frame images determined not to have abnormal symptoms, an abnormality score of, for example, 0 can be generated. This allows for the differentiation between frame images with and without abnormal symptoms. Of course, the abnormality score is not limited to 1 and 0; those skilled in the art can set it according to specific application scenarios, as long as it can distinguish between frame images with and without abnormal symptoms.

[0041] The self-motion of an in-vehicle camera system is a quantitative representation of the system's own motion. It can be represented by the distance the system moves within a preset coordinate system between the capture of two adjacent frames. In one possible implementation, such as... Figure 2As shown, the method for handling unsafe states in a vehicle-mounted camera system according to another embodiment of this disclosure may further include step S500. In step S500, based on the current frame image and the previous frame image, the self-motion of the vehicle-mounted camera system is estimated, and the estimated self-motion is used to perform self-motion compensation on the current frame image to obtain the self-motion-compensated image. Through self-motion compensation, the reliability of the judgment result in step S200 can be further improved.

[0042] In one possible specific implementation, such as Figure 3 As shown, step S500 may include steps S510 to S530.

[0043] In step S510, based on the current frame image and the previous frame image in the frame image, the coarse movement and micro movement of the vehicle-mounted camera system are estimated, wherein the coarse movement indicates a greater range of motion than the micro movement.

[0044] Coarse motion reflects the self-motion of the onboard camera system caused by large vehicle movements such as turning and changing lanes. Micro motion reflects smaller motion amplitudes, such as those caused by smooth or subtle movements of the vehicle. Based on this step, it is possible to effectively estimate larger motion amplitudes caused by vehicle turning and changing lanes, as well as subtle and smooth movements that cannot be captured by coarse motion.

[0045] When estimating coarse movement, the resolution of the frame image can be reduced first to obtain a low-resolution frame image. This can significantly reduce the computational load, thereby improving computational efficiency and meeting real-time requirements.

[0046] In one possible specific implementation, the coarse movement (dx1, dy1) of the vehicle-mounted camera system can be estimated by calculating the correlation between the current frame image and the previous frame image in the low-resolution frame image.

[0047] The following is an example of how to calculate coarse motion. Those skilled in the art should understand that other methods can also be used to calculate larger offsets between images as coarse motion.

[0048] Perform a Fourier transform on the images of the current frame and the previous frame, and calculate the displacement using the phase difference.

[0049] Use the position with the highest peak as the offset.

[0050] The sharper the peak, the higher the confidence level should be.

[0051] In one possible specific implementation, the coarse movement (dx1, dy1) of the vehicle-mounted camera system can be obtained through the following steps:

[0052] (1) Perform Fourier transform on the previous frame image and the current frame image using Formula 1, where the brightness distribution of the image is decomposed into amplitude and phase.

[0053] , Formula 1

[0054] in, Let f(x,y) represent the previous frame image, F(u,v) represent the result of the Fourier transform of the previous frame image, g(x,y) represent the current frame image, and G(u,v) represent the result of the Fourier transform of the current frame image.

[0055] (2) Calculate the normalized cross spectrum using Formula 2, where the Fourier transform results of the two images are multiplied and normalized, and only the phase difference is extracted.

[0056] Formula 2

[0057] (3) Use Formula 3 to perform inverse Fourier transform to restore the original image coordinate system.

[0058] Formula 3

[0059] in, This represents the inverse Fourier transform.

[0060] (4) Detect and read the peak position, and set the position of the highest peak as the offset as in Formula 4.

[0061] Formula 4

[0062] In Formula 4, (Δx, Δy) represents the coarse movement (dx1, dy1) of the vehicle-mounted camera system.

[0063] Additionally, peak height and sharpness can be further measured to calculate reliability. For example, as shown in Formula 5, the PSR (Peak to Sidelobe Ratio) can be used for reliability assessment, or it can be divided by 10 to obtain a normalized numerical reliability W1. This reliability W1 will be used in step S550, mentioned later, to estimate the self-motion of the vehicle-mounted camera system. The reliability can also be determined based on other statistical distributions related to the coarse motion calculation, as long as they reflect the accuracy of the obtained coarse motion.

[0064] Formula 5

[0065] Where p represents the peak height, This represents the average value around the peak (the ring-shaped region). This represents the standard deviation around the peak (the annular region).

[0066] It should be noted that the above is only one possible example implementation for calculating the coarse movement of an in-vehicle camera system. The processing method according to the embodiments of this disclosure is not limited to this, and those skilled in the art can use other methods to calculate the coarse movement of the in-vehicle camera system. As long as the coarse movement of the in-vehicle camera system is used to estimate its self-motion when performing unsafe condition detection of the in-vehicle camera system, it falls within the protection scope of this disclosure.

[0067] In step S510, a lightweight AI model can be used to generate feature maps of the low-resolution frame image, and the correlation between the feature maps of the current frame image and the feature maps of previous frames in the low-resolution frame image can be calculated to estimate the micro-motion (dx2, dy2) of the vehicle-mounted camera system. The correlation between feature maps can be calculated based on optical flow estimation. Therefore, the micro-motion can be calculated from the optical flow estimation of the feature maps, for example, by first generating feature maps using an AI model and then performing optical flow estimation based on the feature maps. A lightweight AI model can refer to an artificial intelligence model that significantly reduces the number of parameters, computational complexity, and resource consumption while maintaining core performance through optimized algorithm design or compression techniques. This reduces power consumption, and this step can be performed on the camera side.

[0068] In one possible implementation, the micro-motion can be obtained through the following approach. Those skilled in the art will understand that even finer, more precise offsets of the image can be calculated as micro-motion using other methods.

[0069] By analyzing the gradient and temporal changes of an image (such as the feature map mentioned above), we can calculate how much a point has moved in which direction.

[0070] The image (such as the feature map mentioned above) is segmented into blocks, and motion vectors are inferred by optical flow estimation for each block.

[0071] The outlier exclusion process described later is implemented based on the motion vectors calculated here.

[0072] The Inlier Ratio (the proportion of vectors in the whole that are consistent with the average motion) is used to calculate whether the background vector is neat and to calculate the reliability W2.

[0073] More specifically, Equation 6 can be used to estimate the micro-movement (dx2, dy2) of the vehicle-mounted camera system:

[0074] Formula 6

[0075] Where I represents image intensity, x and y represent pixel positions, t represents time (frame number), I / x represents the horizontal gradient of the image, I / y represents the vertical gradient of the image, I / t represents the brightness change between frames, and (dx, dy) is the micro-movement of the vehicle camera system (dx2, dy2).

[0076] Furthermore, if the background vector is regularized, the reliability W2 is high. For example, the Inlier Ratio can be used as an indicator of reliability W2. The Inlier Ratio can be calculated as follows: calculate the difference between each block vector and the average vector; the number of differences less than a threshold is denoted as N inlier, and the total number of block vectors is denoted as Ntotal; then, Inlier ratio = N inlier / Ntotal. The reliability W2 mentioned here will be used in step S550 mentioned later when estimating the self-motion of the vehicle-mounted camera system. The reliability can also be determined based on other statistical distributions related to the calculation of micro-motion, as long as they can reflect whether the obtained micro-motion is accurate.

[0077] It should be noted that the above are only a few possible examples of implementations for calculating the coarse and micro movements of the vehicle-mounted camera system. The processing methods according to the embodiments of this disclosure are not limited to these, and those skilled in the art can use other methods to calculate the coarse and micro movements of the vehicle-mounted camera system. As long as the coarse and micro movements of the vehicle-mounted camera system are used to estimate its self-motion when performing unsafe condition detection, it falls within the protection scope of this disclosure.

[0078] In step S520, the statistical distributions of the coarse and micro shifts are calculated to determine their respective reliability W1 and W2. For example, steps S510 and S520 can be performed on each frame image to obtain the coarse and micro shifts when each frame image is used as the current frame image. The statistical distributions of these coarse and micro shifts are then calculated, and the reliability can be determined using the consistency of the distribution and background integration (sharpness of the correlation peak) as indicators. For the specific implementation method of determining reliability W1 and W2, please refer to the description of the estimation processing of coarse and micro shifts above, which will not be repeated here.

[0079] Additionally, it can identify regions with significantly different motion from their surroundings (outlier regions), classifying these regions as moving objects and removing their corresponding vector data. For example, the image can be segmented into blocks, and the correlation coefficient vector of each block can be calculated. By analyzing the vector distribution, regions with vectors that deviate significantly from their surroundings can be identified as moving objects and excluded from self-motion compensation. The presence of moving objects can cause background compensation to be shifted, potentially increasing the overall residual. After self-motion compensation, the portion outside the background will appear as a residual. Within the residual, portions that maintain the same position and shape for an extended period will be detected as stains or water droplets. Therefore, residuals can be used for stain or water droplet detection.

[0080] In one possible implementation, scene classification using a lightweight AI model can also be performed simultaneously, such as daytime, nighttime, backlighting, rain, fog, and ultra-low light. The anomaly detection threshold and reliability values ​​W1 and W2 used in generating anomaly symptom scores can be automatically adjusted based on the scene type. For example, at night, since contrast is inherently reduced, the anomaly detection threshold can be temporarily lowered. Additionally, at night, there are more moving objects, so the reliability W1 for coarse movement can be appropriately increased.

[0081] In step S530, the coarse movement and the micro movement are weighted and summed using the reliability W1 for the coarse movement and the reliability W2 for the micro movement to estimate the self-motion quantity (dx, dy) of the vehicle-mounted camera system. A higher reliability W1 is achieved by emphasizing the coarse movement, while a higher reliability W2 is achieved by emphasizing the micro movement. For example, the self-motion quantity of the vehicle-mounted camera system can be estimated using Equation 7:

[0082] (dx, dy) = W1 × (dx1, dy1) + W2 × (dx2, dy2) Formula 7,

[0083] Where W1 + W2 = 1.

[0084] In this way, the two-stage self-motion compensation based on the (phase) correlation of low-resolution images and the (phase) correlation of AI feature maps suppresses the false correlation caused by self-motion / moving bodies, thus stabilizing the detection.

[0085] In step S500, the estimated self-motion is also used to perform self-motion compensation on the current frame image to obtain the self-motion compensated image. In one possible implementation, the current frame can be shifted in the opposite direction based on the self-motion estimated from the previous frame of the camera. If the estimated camera movement is dx to the right and dy downwards, then the current frame is shifted dx to the left and dy upwards.

[0086] As described above, by removing the moving object and estimating the self-motion using only the background, the non-background portion will form a residual after self-motion compensation. In this residual, portions that maintain the same position and shape for a long time will be detected as stains. Therefore, this residual can be used for stain detection.

[0087] In one possible implementation, such as Figure 2 As shown, the method for handling unsafe conditions in a vehicle-mounted camera system according to another embodiment of this disclosure may further include step S600. In step S600, the difference between the image after motion compensation and the image before motion compensation is extracted as a residual, and pattern recognition is performed using the anomaly detection model based on the image after motion compensation and / or the residual to obtain the pattern recognition result.

[0088] In abnormal situations, such as when foreign objects are attached to the lens, these objects will not follow the lens's own motion model. Therefore, as mentioned above, these areas will stand out in the residuals. Inputting the residuals into an anomaly detection model allows us to determine the type of anomaly in that area. The anomaly detection model can be a lightweight AI model, such as a trained convolutional neural network. By inputting motion-compensated frame images and / or residuals, it can output matching results against predefined anomaly patterns. Anomaly patterns could be, for example, raindrop / water stain patterns, mud / dirt patterns, snow / ice cover patterns, etc. The matching results can include pattern labels and corresponding confidence scores, which can be used to subsequently determine the type of unsafe condition by combining anomaly symptom scoring.

[0089] In one possible implementation, step S200 may include: comprehensively determining whether there is an unsafe state in the vehicle-mounted camera system by weighted fusion of at least two of the abnormal symptom scores, the residuals, and the pattern recognition results.

[0090] One possible implementation of step S200 is as follows: If in step S600, for the current frame image, the anomaly detection model identifies the presence of "raindrops" with high confidence, and a corresponding speckled pattern also appears in the residual of the current frame image, and in step S400, because the image clarity index of the current frame image remains very low for a prolonged period, an anomaly score indicating the presence of anomalies is generated, then the system can determine with great certainty that the vehicle camera system has an unsafe condition of "raindrop obstruction".

[0091] In another possible implementation, if there is a slight residual in the residual of the current frame image, but the anomaly detection model does not identify any known anomaly pattern for the current frame image, and generates an anomaly score for the current frame image that does not have any anomaly signs, then the system may consider the residual as noise and determine that there is no unsafe condition in the vehicle system.

[0092] In addition, depending on environmental factors such as different time periods and weather conditions, some of the above three items can be selected as the basis for judging unsafe conditions. For example, during the daytime, dynamic recognition is highly effective and AI is stable, so the focus is on the combination of residual and AI (i.e., anomaly detection model); at night, because the reliability of residual is reduced, the focus is on the combination of image statistics (i.e., anomaly sign scoring) and AI; and in rainstorm weather, the combination of residual + AI is preferred.

[0093] In another possible implementation, the anomaly score, the residual between the image after motion compensation and the image before motion compensation, and the pattern recognition result can all be converted into corresponding numerical values. These values ​​are then weighted, and calculations are performed based on the weighted values. The results are used to determine if an unsafe state exists. For example, the anomaly score can be represented by 0 or 1. If a portion of the residual maintains the same position and shape for a prolonged period, the corresponding value is 1; otherwise, it is 0. Each detection result of the anomaly detection model can be assigned a corresponding numerical value. For example, 1, 2, and 3 represent the three types, and 0 indicates no type. The weights corresponding to the numerical values ​​can be set as needed. For example, as mentioned above, weights can be set based on time period or environmental factors, or the confidence level of the anomaly detection model's classification result for each category can be used as the weight of the corresponding numerical value. The weighted values ​​can be obtained based on a preset calculation method. This application does not restrict the specific calculation method. For example, the value corresponding to the residual and the value of the classification result of water droplets / dirt output by the anomaly detection model can be weighted and summed as the value corresponding to the water droplet / dirt classification result. The weighted anomaly score can be used as an accumulation term or a multiplication term to control the final classification result.

[0094] Thus, in step S200, by combining all or part of the following three factors—the abnormality score for the current frame image, the residual between the current frame image after self-motion compensation by the vehicle camera system and the image before self-motion compensation, and the pattern recognition result of the current frame image using the anomaly detection model—to determine whether there is an unsafe state in the vehicle camera system, the false alarm rate can be minimized while ensuring a high detection rate.

[0095] The following describes the beneficial effects of determining whether an unsafe state exists in a vehicle-mounted camera system according to embodiments of the present disclosure, in conjunction with existing technology.

[0096] Patent document 1 (CN118247694A) discloses a method for detecting occlusion based on time-series changes in statistical computing block units. The structure of this patent document is as follows:

[0097] ① Divide each frame of the video captured by the camera into multiple image blocks of the same size.

[0098] ② Determine the group of blocks that correspond in spatial position between adjacent frames.

[0099] ③ Calculate the image similarity between corresponding blocks. Specifically, use features such as pixel gradient intensity and angle to quantify the visual consistency between blocks.

[0100] ④ Calculate the rate of change of similarity of each block within a specific time period.

[0101] ⑤ When the rate of change in similarity is low (the appearance of the block hardly changes even if the frame changes), it is determined that the block comes from an obstruction on the lens.

[0102] This method triggers occlusion detection based on minimal image changes between frames. However, when the background image moves significantly due to a vehicle turning or accelerating / decelerating while part of the lens is obscured, the blocks captured from completely different subjects are compared, inevitably reducing the similarity between them. This leads to a decrease in the inferred similarity of the occluded area, making detection impossible. Therefore, Patent Document 1 has the following problems:

[0103] Question 1: The detection sensitivity decreases due to local occlusion caused by self-motion.

[0104] Patent documents 2 (CN112804522B), 3 (US2025046075A1), and 4 (CN112446246B) all involve technologies that use AI models to detect anomalies (dirt, obstructions, etc.) in images from vehicle-mounted cameras. These technologies are characterized by inputting the acquired images into an AI model to determine the presence and type of anomalies such as water, mud, dirt, and overexposure.

[0105] Patent Document 2 is specifically designed for detecting unsafe conditions in stationary states. However, background changes in the image caused by vehicle movement (acceleration, turning) are misidentified as lens dirt or occlusion, leading to a decrease in detection accuracy after the vehicle begins to move. Patent Document 3 uses the unreliability (reliability) estimation results of the AI ​​model itself to reduce the risk of false detection / misjudgment to some extent. However, because it only uses the introspective unreliability estimation results of a single AI model, the risk of misjudgment in unlearned scenarios remains high. Therefore, Patent Documents 2 and 3 have the following problems:

[0106] Question 2: Risk of false detection / false judgment.

[0107] In Patent Document 4, since the entire image is processed pixel by pixel, the requirements for accuracy and detail are very high. However, this high-precision detection requires a large-scale AI model, and if it is always performed in a resource-constrained environment such as an in-vehicle camera / ECU, it will lead to excessive processing load and heat generation. Therefore, Patent Document 4 has the following problems:

[0108] Question 3: Handling the trade-off between load and system.

[0109] According to the processing method of the above embodiments of this disclosure, for images acquired by the camera device of the vehicle-mounted camera system, the self-motion of the camera is compensated based on two phase correlation relationships: coarse correlation and fine correlation. This can remove image changes in appearance caused by camera posture changes or vibrations. In other words, this disclosure suppresses erroneous correlations caused by self-motion / moving bodies through two-level self-motion compensation, thereby stabilizing the detection and solving the aforementioned problem 1.

[0110] Furthermore, the processing method according to the above embodiments of this disclosure uses image statistical information and lightweight AI-based pattern matching and self-motion compensation of low-resolution images to determine unsafe states. The self-motion compensation is characterized by calculation based on the phase correlation of the low-resolution image and the phase correlation of the AI ​​feature map, thus further increasing reliability. By integrating the two, anomalies can be stably detected while reducing false detections. Therefore, this disclosure achieves robustness and suppresses false detections through a combination of statistics (anomaly symptom extraction) + AI (feature map generation / pattern matching) + self-motion compensation, thereby solving the aforementioned problem 2.

[0111] Furthermore, the processing method according to the above embodiments of this disclosure only runs the lightweight AI (high-precision AI is activated only when an anomaly is detected, as will be described later), thus enabling low-power detection of dirt on the camera lens. water droplets This eliminates unsafe conditions such as blurriness, thus resolving problem 3 mentioned above. Therefore, the processing method according to the above embodiments of this disclosure can be performed at the camera side.

[0112] In addition to solving the aforementioned problems 1-3 in the prior art, the method for handling unsafe states in the vehicle-mounted camera system according to the above embodiments of this disclosure also produces the following technical effects compared with the prior art:

[0113] Effect 1: Even at night, in backlight, or in the rain, self-motion compensation is unlikely to fail;

[0114] Effect 2: It can reliably separate dirt and moving objects based on their persistence over time and spatial characteristics;

[0115] Effect 3: Due to its low power consumption, the process can be completed at the camera side, thus eliminating reliance on the ECU and reducing system costs.

[0116] In one possible implementation, the types of unsafe conditions mentioned in step 300 may include: water droplets / dirt adhesion, freezing / blurring, fog / low light. The execution of unsafe condition avoidance procedures may include:

[0117] If the unsafe condition is caused by water droplets / dirt adhesion, the cleaning system will be activated.

[0118] If the type of unsafe condition is frozen / fuzzy, activate the heater;

[0119] When the unsafe condition is fog / low light, a full AI image processing program is initiated to process the frame image using a full AI model to improve accuracy. Here, the full AI model, as opposed to a lightweight AI model, refers to the original AI model that has not undergone compression or other processing. The full AI model can provide high performance to improve image quality.

[0120] In one possible specific implementation, unsafe state avoidance processing can be performed, for example, as shown in Table 1.

[0121] Table 1: Abnormal Avoidance Measures and Notification Content by Category

[0122]

[0123] In one possible implementation, the processing method according to the embodiments of this disclosure may further include: if the normal state cannot be restored after performing the unsafe state avoidance processing, notifying the downstream system of the occurrence of the anomaly and the type of the unsafe state, so as to eliminate the unsafe state as soon as possible.

[0124] To facilitate a better understanding of the technical content of this disclosure, Figure 4 A flowchart illustrating the overall process for handling an unsafe state in a vehicle-mounted camera system according to an embodiment of the present disclosure is provided. Step ① can be found in the description related to step S400; step ② can be found in the description related to step S510; step ③ can be found in the description related to step S510; step ④ can be found in the description related to step S510; steps ⑤⑥⑦⑧ can be found in the description related to steps S520 and S530; step ⑨ can be found in the description related to step S600; and step ⑩ can be found in the description related to step S200. Please refer to the description related to step S300. It will not be repeated here.

[0125] Thus, the processing method according to the above embodiments of this disclosure combines image statistical information with pattern matching and self-motion compensation based on lightweight AI for low-resolution images to determine unsafe conditions. The self-motion of the camera is compensated based on two phase correlation relationships: coarse correlation and micro correlation. Therefore, it is possible to perform high-precision detection with low false recognition rate in real time with low power consumption. This enables timely and accurate detection of unsafe conditions in the vehicle camera system and allows for targeted responses, thereby improving the reliability of the vehicle camera system.

[0126] This disclosure also provides an apparatus for handling unsafe states in a vehicle-mounted camera system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0127] Figure 5 This is a block diagram illustrating an unsafe condition processing device 1900 in a vehicle-mounted camera system according to an exemplary embodiment. For example, device 1900 may be provided as a server or terminal device. (Refer to...) Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0128] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0129] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0130] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0131] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0132] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0133] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0134] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0135] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0139] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for handling unsafe conditions in a vehicle-mounted camera system, characterized in that, include: Acquire frame images continuously captured by the vehicle-mounted camera system; The system determines whether there is an unsafe condition in the vehicle camera system based on the abnormal sign score of the current frame image, the image after self-motion compensation of the vehicle camera system on the current frame image, and the pattern recognition result of the current frame image using the anomaly detection model. as well as In the event of the aforementioned unsafe state, unsafe state avoidance processing is performed according to the type of the unsafe state.

2. The processing method according to claim 1 further includes: For the frame image, the values ​​of one or more indicators reflecting image quality are calculated, and an abnormality score for the current frame image is generated based on whether the values ​​of the one or more indicators are all below or above a specific threshold for a predetermined duration.

3. The processing method according to claim 1 further includes: Based on the current frame image and the previous frame image, the self-motion of the vehicle-mounted camera system is estimated, and the estimated self-motion is used to perform self-motion compensation on the current frame image to obtain the self-motion compensated image.

4. The processing method according to claim 3 further includes: The difference between the image after motion compensation and the image before motion compensation is extracted as the residual; as well as Based on the self-motion compensated image and / or the residual, the anomaly detection model is used to perform pattern recognition to obtain the pattern recognition result.

5. The processing method according to claim 3, estimating the self-motion of the vehicle-mounted camera system includes: Based on the current frame image and the previous frame image in the frame image, the coarse movement and micro movement of the vehicle-mounted camera system are estimated, wherein the coarse movement indicates a greater range of motion than the micro movement. Calculate the statistical distributions of the coarse and micro shifts to determine their respective reliability; and The coarse movement and the micro movement are weighted and summed using the reliability for the coarse movement and the reliability for the micro movement to estimate the self-motion of the vehicle-mounted camera system.

6. The processing method according to any one of claims 1 to 5, determining whether an unsafe state exists in the vehicle-mounted camera system includes: The presence of an unsafe condition in the vehicle-mounted camera system is comprehensively determined by fusing at least two of the following: the abnormal symptom score, the residual between the image after motion compensation and the image before motion compensation, and the pattern recognition result.

7. The processing method according to any one of claims 1 to 5, characterized in that, The types of unsafe conditions include: water droplets / dirt adhesion, freezing / blurring, fog / low light; Among them, the handling of unsafe states includes: If the unsafe condition is caused by water droplets / dirt adhesion, the cleaning system will be activated. If the type of unsafe condition is frozen / fuzzy, activate the heater; When the unsafe condition is of the type of fog / low light, a full AI image processing program is initiated to process the frame image using a full AI model.

8. The processing method according to claim 7, characterized in that, Also includes: If the system fails to return to normal after performing the unsafe state avoidance process, it notifies the downstream system of the occurrence of the anomaly and the type of the unsafe state.

9. A device for handling unsafe conditions in a vehicle-mounted camera system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images