Monitoring method and device for sensing access, medium and equipment

Abstract

The embodiment of the invention discloses a sensing path monitoring method and device, a medium and equipment, and the method comprises the steps: obtaining first original sensor data collected by a first image sensor in a first sensing path; determining first dead pixel detection information based on the first original sensor data; based on the first dead pixel detection information, determining a confidence coefficient of the first sensing path in an aging state; and determining a monitoring result of the first sensing path based on the confidence coefficient. According to the embodiment of the invention, the detection real-time performance of sensing the aging problem of the access can be improved, and the safety of intelligent driving is ensured.

Description

Technical Field

[0001] The present disclosure relates to intelligent driving technology, and in particular to a monitoring method, device, medium and equipment for a sensing path. Background Art

[0002] In the field of intelligent driving, the video perception pathway is the core technology of intelligent driving. The perception pathway refers to the hardware pathway used in the process of acquiring raw sensor data from the image sensor (Sensor) of the camera (or camera) to obtaining image data. Video perception data (i.e., image data), such as YUV (Luminance, Chrominance, brightness and chromaticity) data, is obtained through the perception pathway, and then the video perception data is perceived and processed through the perception algorithm or perception model to obtain the perception result. The perception results are used for intelligent driving planning and control of movable devices (such as vehicles, drones, robots, etc.) to realize intelligent driving functions. With the long-term operation of intelligent driving functions, the perception pathway is prone to hardware aging problems, which can easily affect the safety of intelligent driving. In related technologies, the aging problem of the perception pathway is usually detected by manually connecting to external detection equipment, and this detection method has low real-time performance. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, apparatus, medium and device for monitoring a sensing path, which can detect the aging of the sensing path in a timely manner and improve the real-time performance of discovering aging problems.

[0004] A first aspect of an embodiment of the present disclosure provides a method for monitoring a perception path, comprising: acquiring first raw sensor data collected by a first image sensor in a first perception path; determining first bad pixel detection information based on the first raw sensor data; determining a confidence level that the first perception path is in an aging state based on the first bad pixel detection information; and determining a monitoring result of the first perception path based on the confidence level.

[0005] According to a second aspect of an embodiment of the present disclosure, a monitoring device for a perception channel is provided, comprising: a first processing module for acquiring first raw sensor data collected by a first image sensor in a first perception channel; a second processing module for determining first bad pixel detection information based on the first raw sensor data; a third processing module for determining a confidence level that the first perception channel is in an aging state based on the first bad pixel detection information; and a fourth processing module for determining a monitoring result of the first perception channel based on the confidence level.

[0006] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the monitoring method of the perception path described in any of the above embodiments of the present disclosure.

[0007] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the method for monitoring the perception path described in any of the above embodiments of the present disclosure.

[0008] A fifth aspect of the embodiments of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the method for monitoring the perception path provided by any of the above embodiments of the present disclosure is executed.

[0009] Based on the monitoring method, device, medium and equipment of the perception pathway provided by the above-mentioned embodiments of the present disclosure, the first raw sensor data collected by the first image sensor in the first perception pathway can be obtained, and the first bad pixel detection information can be determined based on the first raw sensor data, and then the confidence that the first perception pathway is in an aging state can be determined based on the first bad pixel detection information, and the monitoring result of the first perception pathway can be determined based on the confidence. Since the aging state of the perception pathway can be monitored based on the raw sensor data collected in real time by the image sensor in the perception pathway, the aging problem can be monitored in time when the aging problem occurs in the perception pathway, thereby improving the real-time detection of the aging problem of the perception pathway and ensuring the safety of intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an exemplary application scenario of the monitoring method of the perception pathway provided by the present disclosure;

[0011] Figure 2 is a flow chart of a method for monitoring a sensing path provided by an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a flow chart of a method for monitoring a sensing path provided by another exemplary embodiment of the present disclosure;

[0013] Figure 4 is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure;

[0014] Figure 5 is a flowchart of a method for monitoring a sensing path provided by another exemplary embodiment of the present disclosure;

[0015] Figure 6is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure;

[0016] Figure 7 is a flowchart of a method for monitoring a sensing path provided by another exemplary embodiment of the present disclosure;

[0017] Figure 8 is a flowchart of a method for monitoring a sensing path provided by an exemplary embodiment of the present disclosure;

[0018] Fig. 9 is a schematic diagram of the structure of a monitoring device for sensing a path provided by an exemplary embodiment of the present disclosure;

[0019] Fig.10 is a structural schematic diagram of a monitoring device for a sensing path provided by another exemplary embodiment of the present disclosure;

[0020] Fig.11 is a structural schematic diagram of a monitoring device for sensing a path provided by yet another exemplary embodiment of the present disclosure;

[0021] Fig.12 is a structural schematic diagram of a monitoring device for sensing a path provided by another exemplary embodiment of the present disclosure;

[0022] Fig.13 is a structural schematic diagram of a monitoring device for sensing a path provided by yet another exemplary embodiment of the present disclosure;

[0023] Fig.14 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0025] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0026] SUMMARY OF THE DISCLOSURE

[0027] In the process of realizing the present disclosure, the inventors found that in the field of intelligent driving, the video perception path is the core technology of intelligent driving. The perception path refers to the hardware path used in the process of acquiring raw sensor data from the image sensor (Sensor) of the camera (or camera) to obtaining image data. For example, the perception path may include image sensors, various signal processing units in the image signal processing (Image Signal Processing, referred to as: ISP) module, etc. Video perception data, such as YUV (Luminance, Chrominance, brightness and chromaticity) data, are obtained through the perception path, and then the video perception data is perceived and processed through the perception algorithm or perception model to obtain the perception result. The perception result may include the perception result of lane lines, obstacles, etc. The perception result is used for intelligent driving planning and control of movable devices (such as vehicles, drones, robots, etc.) to realize the intelligent driving function. With the long-term operation of the intelligent driving function, the perception path is prone to hardware aging problems, which easily affect the safety of intelligent driving. In the related art, the aging problem of the perception path is usually detected by manually accessing external detection equipment, and this detection method has low real-time performance.

[0028] Exemplary Overview

[0029] Figure 1 This is an exemplary application scenario of the monitoring method of the perception channel provided by the present disclosure. Figure 1As shown, in the intelligent driving scenario, a perception path 11, a perception system (or perception function component) 12, and a planning and decision system (or planning and decision function component) 13 may be provided on the mobile device. The perception system 12 and the planning and decision system 13 may correspond to a chip respectively, or may be integrated into a chip. Taking the perception system as an example, the perception function software (i.e., the perception function application program) may be executed by the processor (Central Processing Unit, referred to as CPU) in the chip, and the processing process of the perception algorithm or model in the perception function software may be accelerated by combining one or more hardware in the neural network processor, graphics processor (Graphics Processing Unit, referred to as GPU), etc. in the chip, so that the software and hardware cooperate to realize the perception function. The neural network processor may include, for example, at least one of NPU (Neural Network Processing Unit), VDPS (Vision Digital Signal Processor), etc. The monitoring device (referred to as monitoring device) 14 of the perception path disclosed in the present invention may be coupled to the perception path 11, and the monitoring device 14 monitors the perception path 11 by executing the monitoring method of the perception path disclosed in the present invention. The monitoring device 14 may be at least one of any processor, controller, chip or other device with corresponding processing functions on the mobile device. The perception path 11 may include one or more hardware components (or hardware modules) such as an image sensor 111 and an image signal processing module 112. The image signal processing module 112 may include a black level correction unit, a bad pixel detection unit, a lens shading correction unit, a white balance unit, a gamma correction unit, a color correction unit, a color space conversion unit, a chromaticity processing unit, a brightness processing unit, an image format conversion unit and other signal processing units. Optionally, the perception path 11 may include one or more hardware components in each signal processing unit in the image sensor 111 and the image signal processing module 112. The image signal processing module 112 performs image signal processing on the raw sensor data collected by the image sensor 111 to obtain image data, and the perception system 12 performs perception processing on the image data to obtain a perception result. The planning and decision system 13 may perform planning and decision-making based on the perception result, obtain a control instruction of the mobile device, and control the driving state of the mobile device based on the control instruction to realize the intelligent driving function. The movable device may include one or more sensing pathways 11, such as a sensing pathway for sensing one or more perspectives of the external environment of the movable device and a sensing pathway for sensing one or more perspectives of the environment inside the cabin of the movable device.When the intelligent driving function is activated, the perception pathway 11 is activated accordingly, and the monitoring device 14 of the perception pathway is connected to the perception pathway 11 to execute the monitoring method of the perception pathway disclosed in the present invention to achieve effective monitoring of the perception pathway 11. Specifically, any perception pathway 11 can be used as the first perception pathway, and the first original sensor data collected by the first image sensor 111 in the first perception pathway is obtained. Based on the first original sensor data, the first bad pixel detection information is determined, and then based on the first bad pixel detection information, the confidence that the first perception pathway is in an aging state is determined, and the monitoring result of the first perception pathway is determined based on the confidence. Since the aging state of the perception pathway 11 can be monitored based on the original sensor data collected in real time by the image sensor 111 in the perception pathway 11, the aging problem can be monitored in time when the perception pathway 11 has an aging problem, thereby improving the real-time detection of the aging problem of the perception pathway 11 and ensuring the safety of intelligent driving.

[0030] It should be noted that Figure 1 This is only an exemplary application scenario. In actual applications, the monitoring method of the perception pathway of the disclosed embodiment is not limited to intelligent driving scenarios, but can also be applied to any other scenarios with perception pathways, such as intelligent cockpit scenarios, security monitoring scenarios, and the like.

[0031] Exemplary Methods

[0032] Figure 2 is a flow chart of a method for monitoring a sensing path provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as a vehicle-mounted computing platform, such as Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:

[0033] Step 210: Acquire first raw sensor data collected by a first image sensor in a first sensing channel.

[0034] Among them, the first perception path can be a perception path corresponding to any camera. For example, in an intelligent driving scenario, the first perception path can be a perception path corresponding to any viewing angle camera on a movable device for sensing the external environment. In an intelligent cockpit scenario, the first perception path can be a perception path corresponding to any viewing angle camera in the cockpit of a movable device. The first perception path can include an image sensor (i.e., a first image sensor), an image signal processor (i.e., an image signal processing module, which can be referred to as a first image signal processor), etc. The first raw sensor data is the unprocessed raw data (i.e., RAW data) collected by the first image sensor.

[0035] Step 220: Determine first bad pixel detection information based on the first original sensor data.

[0036] The first bad pixel detection information is the bad pixel information of the first image sensor obtained by detection, and the bad pixel refers to the phenomenon that the pixel information (or pixel value) on the image sensor is inaccurate due to factors such as process defects and signal conversion errors. The first bad pixel detection information may include the number of bad pixels, the position information of each bad pixel, etc. The position information of the bad pixel may include the coordinates of the bad pixel in the image coordinate system corresponding to the first image sensor, and the image coordinate system is a coordinate system with the upper left corner of the image as the origin.

[0037] In some optional embodiments, the first bad pixel detection information can be determined by a bad pixel detection unit in the first sensing path. For example, the first original sensor data is transmitted to the bad pixel detection unit in the first sensing path to obtain the first bad pixel detection information. That is, the first bad pixel detection information can be obtained from the bad pixel detection unit in the first sensing path.

[0038] In some optional embodiments, the first bad pixel detection information may be determined based on the first original sensor data and the historical original sensor data at the historical moment, that is, the first bad pixel detection information is determined by combining at least two frames of original sensor data to improve the accuracy of the first bad pixel detection information.

[0039] Step 230: Determine the confidence level that the first sensing path is in an aging state based on the first bad pixel detection information.

[0040] Among them, aging refers to the phenomenon that the performance of the first perception pathway is gradually reduced due to irreversible damage caused by long-term use. The aging state refers to the state in which the first perception pathway is damaged to a certain extent. The confidence of the aging state refers to the credibility of the first perception pathway being in the aging state. Because bad pixels are not necessarily caused by physical damage to the first perception pathway, but may also be caused by software, white noise and other reasons, in the case of bad pixels, it is necessary to further determine the confidence of the first perception pathway being in the aging state based on the first bad pixel detection information, so as to more accurately judge whether the first perception pathway is in the aging state.

[0041] In some optional embodiments, the first bad pixel detection information can characterize the number and distribution of bad pixels, and the bad pixels caused by aging of the perception path usually have certain characteristics, such as the bad pixels are relatively concentrated. Confidence rules can be set according to the characteristics of the bad pixels caused by aging, and the confidence that the first perception path is in an aging state is determined according to the confidence rules. Specific confidence rules can be set according to actual needs. For example, the confidence rules may include bad pixel concentration condition rules, multi-path perception path verification rules, etc. The bad pixel concentration condition rule is a rule that determines the confidence that the first perception path is in an aging state based on the concentration of bad pixels in the first bad pixel detection information. The multi-path perception path verification rule is a rule that verifies the first bad pixel detection information through the bad pixel detection information of other perception paths that have overlapping viewing angles with the first perception path, and then determines the confidence that the first perception path is in an aging state.

[0042] Step 240: Determine the monitoring result of the first sensing path based on the confidence level.

[0043] In some optional embodiments, the confidence level may be determined as the monitoring result of the first perception path. Alternatively, the confidence level may be further processed to obtain the monitoring result of the first perception path. For example, the monitoring result may include the result of whether the first perception path has an aging problem. A confidence threshold may be set, and when the confidence level is higher than the confidence threshold, the monitoring result is determined to be that the first perception path has an aging problem, otherwise the monitoring result is determined to be that the first perception path does not have an aging problem or does not have an aging problem temporarily, and the monitoring is continued to determine whether there is an aging problem at a subsequent time.

[0044] The monitoring method of the perception path provided in this embodiment can monitor the aging status of the perception path based on the original sensor data collected in real time by the image sensor in the perception path. When the aging problem occurs in the perception path, the aging problem can be monitored in time, thereby improving the real-time detection of the aging problem of the perception path and ensuring the safety of intelligent driving.

[0045] Figure 3 It is a flowchart of a method for monitoring a sensing path provided by another exemplary embodiment of the present disclosure.

[0046] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, the step 230 of determining the confidence that the first sensing path is in an aging state based on the first bad pixel detection information may include the following steps:

[0047] Step 2310, determining the number of bad pixels and bad pixel location information based on the first bad pixel detection information.

[0048] The number of bad pixels is the number of pixels whose pixel values ​​are inaccurate. The bad pixel position information includes the coordinates of each bad pixel in the image coordinate system.

[0049] Step 2320, determining bad pixel distribution information based on the bad pixel quantity and bad pixel location information.

[0050] The bad pixel distribution information may include the variance or mean square error of the position distribution of each bad pixel on the image. The bad pixel distribution information may be determined based on the number of bad pixels and the position of each bad pixel in combination with the variance or mean square error statistical formula.

[0051] Step 2330, determining the bad pixel concentration state according to the bad pixel distribution information.

[0052] The bad pixel concentration state may include at least two states, for example, the bad pixel concentration state may include concentrated and dispersed states. Alternatively, more concentration states of different degrees may be set according to the size relationship of the bad pixel distribution information, for example, dispersed, slightly concentrated, moderately concentrated, highly concentrated, etc. The number of states of the specific bad pixel concentration state is not limited.

[0053] In some optional embodiments, the concentration state of the bad pixels can be determined based on the distribution information threshold. For example, when the bad pixel distribution information is expressed as variance, the distribution information threshold is the variance threshold. When the variance of the bad pixel distribution is less than the variance threshold, the bad pixel concentration state can be determined to be concentrated. When the variance of the bad pixel distribution is greater than the variance threshold, the bad pixel concentration state can be determined to be dispersed (or not concentrated). When the bad pixel distribution information is expressed as mean square error, the distribution information threshold is the mean square error threshold.

[0054] Step 2340: Determine the confidence level that the first sensing path is in an aging state based on the bad pixel concentration state.

[0055] Among them, a mapping rule between the bad pixel concentration state and the confidence can be preset, and the bad pixel concentration state is mapped to the confidence that the first perception path is in an aging state according to the mapping rule. For example, corresponding confidence values ​​can be set for different concentration levels, and then the confidence that the first perception path is in an aging state can be determined according to the specific concentration level of the bad pixel concentration state and the preset confidence values ​​corresponding to the different concentration levels. For another example, the confidence that the first perception path is in an aging state can be determined by combining the bad pixel concentration state and the bad pixel concentration state at a historical moment.

[0056] In this implementation, the bad pixel concentration state is determined based on the first bad pixel detection information, which can effectively characterize whether the perception path has an aging problem. Usually, the bad pixels caused by aging problems are relatively concentrated, and scattered bad pixels may often be just dynamic noise. Based on this, according to the bad pixel concentration state, the confidence that the first perception path is in an aging state can be effectively determined.

[0057] In some optional embodiments, determining the confidence level of the first sensing path being in an aging state according to the bad pixel concentration state in step 2340 may include:

[0058] In response to the bad pixel concentration state being a first state, second bad pixel detection information of at least one second perception channel having an overlapping viewing angle with the first perception channel is determined; based on the second bad pixel detection information of at least one second perception channel, the bad pixel quantity and bad pixel position information corresponding to the first bad pixel detection information are verified to obtain a verification result; based on the bad pixel concentration state and the verification result, the confidence that the first perception channel is in an aging state is determined.

[0059] Among them, the first state may refer to concentration, that is, the variance of the bad pixel distribution is less than the variance threshold, or the mean square error of the bad pixel distribution is less than the mean square error threshold. The bad pixel concentration state is the first state, which means that the bad pixel distribution is relatively concentrated, and it may be a bad pixel caused by aging problems. The second perception path is a perception path other than the first perception path that has an overlapping perspective with the first perception path. The second bad pixel detection information of the second perception path is obtained in a similar way to the first bad pixel detection information, which will not be repeated here. For example, in an intelligent driving scenario, the perception path of the front view (the first perception path) has an overlapping perspective with the perception path of the left front view, and the perception path of the left front view can be used as the second perception path. Similarly, if the perception path of the right front view has an overlapping perspective with the perception path of the front view, the perception path of the right front view can also be used as the second perception path. The number of bad pixels and the bad pixel position information of the first bad pixel detection information are verified based on the second bad pixel detection information of the second perception path to determine whether each bad pixel in the first bad pixel detection information is indeed a bad pixel. In this case, for each second perception channel, based on the bad pixels distributed in the overlapping visual angle in the second bad pixel detection information of the second perception channel, the bad pixels distributed in the overlapping visual angle in the first bad pixel detection information are verified. For each bad pixel position distributed in the overlapping visual angle in the first bad pixel detection information, if the second bad pixel detection information of the second perception channel is also a bad pixel at the position corresponding to the bad pixel position, it is possible that the first perception channel and the second perception channel are both white noise at the bad pixel position, resulting in the first perception channel and the second perception channel uniformly identifying it as a bad pixel, which is not a real bad pixel. If the second perception channel does not detect a bad pixel at the position corresponding to the bad pixel position, and only the first perception channel detects a bad pixel, it can be indicated that the first perception channel is indeed a bad pixel at the bad pixel position. By verifying the number of bad pixels and the bad pixel position information corresponding to the first bad pixel detection information, the white noise points can be filtered out. The verification result may include the result of whether each bad pixel position in the first bad pixel detection information is a bad pixel. Further combined with the bad pixel concentration state and the verification result, the confidence that the first perception channel is in an aging state is determined.

[0060] In some optional embodiments, the aging state can be scored based on the bad pixel concentration state and the verification result, respectively, to obtain a score corresponding to the bad pixel concentration state and a score corresponding to the verification result, and the two scores are weighted according to a preset weight to obtain the confidence that the first sensing path is in the aging state. A scoring rule can be set according to the correlation between the bad pixel concentration state and the verification result and the confidence of the aging state, respectively, and the score corresponding to the bad pixel concentration state and the verification result can be determined based on the scoring rule.

[0061] In this embodiment, when it is determined that the bad pixel concentration state is concentrated, the first bad pixel detection information is verified through the bad pixel detection system information of the second perception channel that has an overlapping viewing angle with the first perception channel, and the verification result is used together with the bad pixel concentration state to determine the confidence level, which can further improve the accuracy of the confidence level.

[0062] In some optional embodiments, determining the confidence level that the first sensing path is in an aging state according to the bad pixel concentration state and the verification result may include:

[0063] In response to the verification result satisfying the first condition, image data corresponding to the first original sensor data is determined; based on the image data, a quality score corresponding to the image data is determined; and according to the bad pixel concentration state, the verification result and the quality score, the confidence that the first perception path is in an aging state is determined.

[0064] Among them, the first condition may refer to the verification result that the number of bad pixels that are indeed bad pixels in the first bad pixel detection information exceeds the number threshold, or the ratio of the number of bad pixels that are indeed bad pixels in the first bad pixel detection information exceeds the ratio threshold. The image data corresponding to the first raw sensor data is image data of a preset format converted by image signal processing. The preset format may be, for example, a YUV (Luminance, Chrominance, brightness and chromaticity) format, an RGB (Red, Green, Blue, red, green, blue color mode) format, etc. That is, the image data may be a YUV image, an RGB image, etc.

[0065] In some optional embodiments, the image data corresponding to the first raw sensor data may be determined by an image signal processing module in the first perception path.

[0066] In some optional embodiments, the quality score corresponding to the image data can be determined by an image quality evaluation algorithm. The image quality evaluation algorithm can adopt any implementable algorithm. For example, the image data can be scanned by a pre-trained quality evaluation model to obtain a quality score corresponding to the image data. The quality evaluation model can be, for example, a model based on the NIQE (Natural Image Quality Evaluator) algorithm, the IL-NIQE (Improved Natural Image Quality Evaluator) algorithm, and the like.

[0067] In some optional embodiments, after obtaining the quality score corresponding to the image data, the confidence that the first perception path is in an aging state can be determined in combination with the bad pixel concentration state, the verification result and the quality score, so as to further improve the accuracy of the confidence. Specifically, the bad pixel concentration state, the verification result and the quality score can be set to score the aging state respectively, and the score corresponding to the bad pixel concentration state is determined according to the scoring rule corresponding to the bad pixel concentration state, the score corresponding to the verification result is determined according to the scoring rule corresponding to the verification result, and the score corresponding to the quality score is determined according to the scoring rule corresponding to the quality score. The score corresponding to the bad pixel concentration state, the score corresponding to the verification result and the score corresponding to the quality score are weighted according to the preset weights to obtain a comprehensive score as the confidence that the first perception path is in an aging state. The scoring rules of each factor (bad pixel concentration state, verification result, quality score) can be set according to the correlation between each factor and the aging state. For example, for the bad pixel concentration state, the scoring rules can be set according to the principle that the more concentrated the bad pixels are, the higher the score is, and the more dispersed the bad pixels are, the lower the score is. For the verification result, the scoring rule can be set according to the principle that the more the number of indeed bad pixels (or the higher the proportion of indeed bad pixels), the higher the score, and vice versa.

[0068] In this embodiment, when the verification result satisfies the first condition, the confidence that the first perception pathway is in an aging state can be further determined in combination with the quality score of the image data corresponding to the first original sensor data, which can further improve the accuracy of the confidence.

[0069] Figure 4 It is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0070] In some optional embodiments, the method of the embodiment of the present disclosure may further include:

[0071] Step 310: Determine image data corresponding to the first original sensor data.

[0072] Step 320: Determine a quality score corresponding to the image data based on the image data.

[0073] The specific operations of step 310 and step 320 may refer to the above-mentioned embodiment.

[0074] Determining the confidence level of the first sensing path being in an aging state according to the bad pixel concentration state in step 2340 may include:

[0075] Step 23410, determining the confidence level that the first perception path is in an aging state based on the bad pixel concentration state and the quality score.

[0076] Among them, scoring rules corresponding to the bad pixel concentration state and the quality score can be set, and the scores corresponding to the bad pixel concentration state and the quality score can be determined according to the scoring rules. The two scores are weighted to obtain the confidence that the first perception channel is in an aging state.

[0077] In this embodiment, the confidence level that the first sensing channel is in an aging state is determined by combining the bad pixel concentration state and the quality score of the image data, which can further improve the accuracy of the confidence level.

[0078] In some optional embodiments, based on the image data, determining the quality score corresponding to the image data may further include: downsampling the image data to obtain target image data of a preset size, the preset size being smaller than the original size of the image data; and determining the quality score corresponding to the image data based on the target image data of the preset size. By downsampling, the amount of image data is reduced, thereby improving processing efficiency.

[0079] In some optional embodiments, the image data may be downsampled by a hardware pyramid module in the perception system to improve the downsampling efficiency.

[0080] Figure 5 It is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0081] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 5 As shown, the step 230 of determining the confidence level that the first sensing path is in an aging state based on the first bad pixel detection information may include:

[0082] Step 23a0, based on the first bad pixel detection information and the historical bad pixel detection information of at least one historical moment of the first sensing channel, determine the confidence that the first sensing channel is in an aging state.

[0083] Among them, the moment when the first original sensor data is collected is called the current moment, and the historical moment is the moment before the current moment. The historical bad pixel detection information of the historical moment is obtained during the image signal processing of the historical moment. A certain time window can be set to cache the historical bad pixel detection information of one or more historical moments before the current moment, and at the current moment, the historical bad pixel detection information of at least one historical moment can be directly obtained from the cache area. After obtaining the first bad pixel detection information at the current moment, the first bad pixel detection information can be cached, and the earliest historical bad pixel detection information of the historical moment can be deleted according to the time window, and so on, and the most recent one or more historical bad pixel detection information is always cached.

[0084] In some optional embodiments, the bad pixel distribution of the first bad pixel detection information and the bad pixel distribution of the historical bad pixel detection information are combined to determine the confidence that the first perception path is in an aging state. For the same pixel position of the same perception path, if a bad pixel is detected at a historical moment, if a bad pixel is still detected at the current moment, it can be determined that the perception path is likely to be damaged due to aging at the pixel position. Therefore, combining the bad pixel detection information at multiple moments can effectively determine the confidence that the first perception path is in an aging state.

[0085] In this embodiment, the confidence that the first sensing channel is in an aging state is determined by combining the first bad pixel detection information and the historical bad pixel detection information of at least one historical moment of the first sensing channel, which can further improve the accuracy of the confidence.

[0086] Figure 6 It is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0087] In some optional embodiments, based on any of the above embodiments, Figure 6 As shown, the method of the embodiment of the present disclosure may also include:

[0088] Step 250 , in response to the monitoring result satisfying a preset condition, determining a target pixel area corresponding to the aging state.

[0089] Among them, if the monitoring result is a confidence level, the preset condition is that the confidence level is greater than the confidence level threshold. If the monitoring result is the result of whether the first perception pathway has an aging problem, the preset condition is that the first perception pathway has an aging problem. If the monitoring result satisfies the preset condition, it means that the confidence level is greater than the confidence level threshold or the monitoring result is that the first perception pathway has an aging problem. When it is determined that the monitoring result satisfies the preset condition, the target pixel area corresponding to the aging state can be further determined. The target pixel area is the area where bad pixels (or bad pixels that are indeed bad pixels rather than white noise after verification, referred to as verified bad pixels) are distributed. The target pixel area can be determined according to the coordinates (or pixel coordinates) of the bad pixels (or verified bad pixels) in the image coordinate system. The pixel points corresponding to each bad pixel (or verified bad pixel) constitute the target pixel area.

[0090] Step 260 : In response to the target pixel area being located in the first preset area range, determine image cropping information according to the target pixel area.

[0091] Among them, the first preset area range is a preset image corner area range, and the first preset area range can be determined according to the degree of influence on the perception result in the viewing angle range corresponding to the first image sensor. The area range that has less influence on the perception result is set to the first preset area range. If the target pixel area is within the first preset area range, it means that the current aging part has little influence on the perception result, and the current perception result can continue to be maintained in downstream applications. For example, the perception result can continue to be used for planning and decision-making to maintain intelligent driving functions, intelligent cockpit functions, etc. In order to avoid the adverse effects of the pixel values ​​of the aging part in the image data on the perception results, the image cropping information can be determined according to the target pixel area, and the image cropping information includes the cropped area information in the image data.

[0092] Step 270, sending the image cropping information to the perception system, so that the perception system crops the image according to the image cropping information, and obtains a perception result based on the cropped image.

[0093] The perception system is a downstream system of the first perception pathway, and the perception system is used to perform perception processing on the image data output by the first perception pathway based on a preconfigured perception algorithm or model to obtain a perception result. After determining the image cropping information, the image cropping information is sent to the perception system, so that the perception system can crop the image (i.e., the image data output by the first perception pathway to the perception system) according to the image cropping information, crop the target pixel area of ​​the aging part, and obtain a cropped image. The perception system performs perception processing based on the cropped image to obtain a perception result.

[0094] In this embodiment, when the aging problem of the first perception path is monitored, the target pixel area of ​​the aged part is determined, and then the image cropping information is determined according to the target pixel area, and provided to the downstream perception system, so that the perception system can crop the aged part in the image before performing perception processing, which can avoid the abnormal pixel value of the aged part from having an adverse effect on the perception result, thereby ensuring the validity of the perception result.

[0095] In some optional embodiments, such as Figure 6 As shown, the method of the embodiment of the present disclosure may also include:

[0096] Step 280 , in response to the target pixel area being located in the second preset area range, determining the driving state of the movable device where the first sensing path is located.

[0097] Among them, the second preset area range is an important viewing angle range that has a great influence on the perception result. For example, the second preset area range is the area range of the middle part of the image, which affects the recognition of dynamic and static objects around the movable device, and thus affects the driving safety of the movable device. If the target pixel area is located in the second preset area range, it means that the abnormal pixel value of the aging part will seriously affect the driving safety of the movable device, and it is necessary to further determine the driving state of the movable device where the first perception path is located. The driving state can include two states: static and moving.

[0098] Step 290, executing the processing instruction corresponding to the driving state.

[0099] Different driving states may correspond to different processing instructions. For example, a stationary state corresponds to one processing instruction, and a moving state corresponds to another processing instruction. The processing instructions corresponding to different driving states may be set according to the safety of the movable device in different driving states.

[0100] In some optional embodiments, the execution of processing instructions corresponding to the driving status of step 290 may include: in response to the driving status being motion, outputting a prompt message to suspend the use of the intelligent driving function; or, in response to the driving status being stationary, sending aging notification information to a server corresponding to the movable device.

[0101] Among them, when the driving state is a moving state, since the aging part has affected the driving safety of the mobile device, the processing instruction corresponding to the movement can be to output a prompt message for suspending the use of the intelligent driving function. Therefore, in response to the driving state being moving, the processing instruction corresponding to the movement is executed, that is, the prompt message for suspending the use of the intelligent driving function is output to prompt the user of the mobile device to take over the mobile device in time and stop using the intelligent driving function to ensure the driving safety of the mobile device. When the driving state is a stationary state, it means that the mobile device is not driving, and the processing instruction corresponding to the stationary state can be executed, that is, to send an aging notification message to the server corresponding to the mobile device. The server corresponding to the mobile device can be a server of a manufacturer connected to the mobile device. The mobile device can establish a connection with the server in a specified manner and send an aging notification message to the server, so that the relevant personnel of the manufacturer can repair or replace the first perception path of the mobile device in time to ensure the subsequent driving safety of the mobile device.

[0102] In this embodiment, when any sensing path of the mobile device has a serious aging problem and the mobile device is moving, a prompt message for suspending the use of the intelligent driving function can be output in time to prompt the user of the mobile device to take over the mobile device in time and stop using the intelligent driving function to ensure the driving safety of the mobile device. When the mobile device is stationary, an aging notification message can be sent to the server corresponding to the mobile device so that the relevant personnel of the manufacturer can repair or replace the first sensing path of the mobile device in time to ensure the subsequent driving safety of the mobile device.

[0103] Figure 7 It is a flowchart of a method for monitoring a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0104] In some optional embodiments, based on any of the above embodiments, Figure 7 As shown, the method of the embodiment of the present disclosure may also include:

[0105] Step 410, obtaining an update data packet of the monitoring method for the sensing path from the server.

[0106] The update data packet may include at least one of the new parameter value corresponding to the parameter to be updated, the new condition corresponding to the condition to be updated, the new rule corresponding to the rule to be updated, etc. in the current monitoring method of the sensing path, and the parameter to be updated may include, for example, one or more thresholds of the above-mentioned variance threshold, mean square error threshold, confidence threshold, etc. The condition to be updated may include, for example, one or more of the above-mentioned preset condition, first condition, etc. The rule to be updated may include, for example, one or more of the above-mentioned confidence rule, mapping rule of bad pixel concentration state and confidence, various scoring rules, etc.

[0107] In some optional embodiments, the relevant staff of the manufacturer may analyze the historical monitoring results of the monitoring method of the sensing path of the embodiment of the present disclosure on one or more mobile devices and the actual state of the corresponding sensing path, and determine the parameters to be updated of the monitoring method of the sensing path and the new parameter values ​​corresponding to the parameters to be updated, the conditions to be updated and the corresponding new conditions, the rules to be updated and the corresponding new rules, etc. For example, for the confidence threshold, if during the multiple monitoring processes of the sensing path, when the confidence is greater than the confidence threshold (such as 0.8 or 80%), the monitoring determines that the sensing path has an aging problem, and the sensing path is actually tested and verified to have an aging problem, the confidence threshold can be increased, such as increasing the confidence threshold from 0.8 to 0.85. The new conditions corresponding to the conditions to be updated and the new rules corresponding to the rules to be updated can be obtained by analyzing and summarizing the historical monitoring results and the corresponding actual detection results.

[0108] In some optional embodiments, the update data package may be downloaded from the server via OTA (Over-the-Air Technology).

[0109] Step 420, updating the monitoring method of the sensing path according to the update data packet, obtaining an updated monitoring method of the sensing path, and using it to monitor the first sensing path.

[0110] After the update data packet is obtained, the monitoring method of the perception path can be updated according to the update data packet, and the updated monitoring method of the perception path can be obtained for continuing to monitor the first perception path. The update can be performed according to the update method supported by the removable device. For example, during the startup process of the removable device, or during the startup process of the chip where the software package corresponding to the monitoring method of the perception path is located, or at other possible times, the update of the monitoring method of the perception path is triggered.

[0111] This embodiment obtains an update data packet of the monitoring method of the perception path from the server to update the monitoring method of the perception path, thereby continuously optimizing the monitoring method of the perception path and further improving the accuracy of the monitoring results of the aging problem of the perception path.

[0112] In some optional embodiments, Figure 8 FIG. 1 is a flowchart of a method for monitoring a sensing path provided by an exemplary embodiment of the present disclosure. Figure 8 As shown, the process of the method may include the following steps:

[0113] Step 4310, camera connection.

[0114] In step 4320, the image sensor of the camera (ie, the first image sensor in the first sensing path) collects raw sensor data (ie, first raw sensor data).

[0115] Step 4330: Hardware performs bad pixel detection, that is, the hardware bad pixel detection unit in the first sensing path performs bad pixel detection to obtain first bad pixel detection information.

[0116] Step 4340, determine the number of bad pixels and bad pixel location information.

[0117] Step 4350, determine bad pixel distribution information.

[0118] Step 4360, determine the bad pixel concentration state.

[0119] Step 4370 , determining whether the bad pixel concentration state is the first state. In response to the bad pixel concentration state being the first state, the process proceeds to step 4380 and step 4410 .

[0120] Step 4380 , determining second bad pixel detection information of at least one second perception channel. The second perception channel is another perception channel having an overlapping viewing angle with the first perception channel. Go to step 4390 .

[0121] Step 4390, verify the number of bad pixels and the bad pixel location information of the first bad pixel detection information. That is, based on the second bad pixel detection information of at least one second sensing channel, verify the number of bad pixels and the bad pixel location information of the first bad pixel detection information to obtain a verification result. Go to step 4510.

[0122] Step 4410 , determine the image data corresponding to the original sensor data. Go to step 4420 .

[0123] Step 4420 , determine the quality score corresponding to the image data. Go to step 4510 .

[0124] Step 4510 , determining the confidence level that the first sensing path is in an aging state based on the bad pixel concentration, the verification result and the quality score.

[0125] Step 4520 , determine the monitoring result of the first sensing pathway. Go to step 4530 .

[0126] Step 4530: determine whether the monitoring result meets the preset condition. In response to the monitoring result meeting the preset condition, go to step 4540.

[0127] Step 4540 , determining the target pixel region corresponding to the aging state. Go to step 4550 .

[0128] Step 4550 , determine whether the target pixel region is located in the first preset region range. If yes (ie the target pixel region is located in the first preset region range), go to step 4560 ; if no (ie the target pixel region is not located in the first preset region range), go to step 4610 .

[0129] Step 4560 , determine image cropping information and proceed to step 4570 .

[0130] Step 4570, notifying the perception system to perform image cropping, that is, sending image cropping information to the perception system.

[0131] Step 4610, determining the driving status of the movable device.

[0132] Step 4620: In response to the driving state being in motion, output a prompt message for suspending the use of the intelligent driving function.

[0133] Step 4630, in response to the driving state being stationary, sending aging notification information to the server corresponding to the movable device.

[0134] The specific operations of the above steps can be found in the above embodiments and will not be described in detail here.

[0135] The disclosed embodiment realizes the automatic inspection of the aging problem of the perception pathway by combining hardware bad point detection and software image quality evaluation, thereby improving the real-time detection of aging problems. Moreover, the effective monitoring of the perception pathway can be realized to the greatest extent through the inherent capabilities of the chip where the perception pathway is located through full coordination verification of multiple perception pathways, thereby avoiding or reducing the increase of hardware. In the case of monitoring the aging problem of the perception pathway, it can be flexibly handled according to the driving state of the mobile device, and the driving experience of the user (such as the driver) of the mobile device can be improved while ensuring the safety of the mobile device. Moreover, by automatically monitoring the aging problem of the perception pathway, the manufacturer can quickly and accurately locate the aging perception pathway, reduce the manufacturer's maintenance cost, improve the maintenance work efficiency, and reduce labor costs. In addition, since the output results of the intelligent driving perception pathway cannot be displayed intuitively, the relevant technology needs to use external equipment to detect the hardware aging problem of the perception pathway. The method of the disclosed embodiment effectively solves this problem of the relevant technology. The method of the disclosed embodiment can avoid using external equipment, and can quickly identify the hardware aging problem of the perception pathway based on the function of the intelligent driving system itself.

[0136] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.

[0137] Any of the sensing pathway monitoring methods provided in the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any of the sensing pathway monitoring methods provided in the embodiments of the present disclosure can be executed by a processor, such as the processor executing any of the sensing pathway monitoring methods mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0138] Exemplary Devices

[0139] Fig. 9 is a schematic diagram of a monitoring device for sensing a path provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, such as Fig. 9 The device shown may include: a first processing module 51 , a second processing module 52 , a third processing module 53 and a fourth processing module 54 .

[0140] The first processing module 51 is used to obtain first original sensor data collected by the first image sensor in the first sensing channel.

[0141] The second processing module 52 is configured to determine first bad pixel detection information based on the first original sensor data.

[0142] In some optional embodiments, the second processing module 52 may be a bad pixel detection unit in the first sensing path, or the second processing module 52 may obtain the first bad pixel detection information corresponding to the first original sensor data from the bad pixel detection unit in the first sensing path.

[0143] The third processing module 53 is used to determine the confidence level that the first sensing path is in an aging state based on the first bad pixel detection information.

[0144] The fourth processing module 54 is used to determine the monitoring result of the first perception channel based on the confidence level.

[0145] Fig.10 It is a structural diagram of a monitoring device for a sensing path provided by another exemplary embodiment of the present disclosure.

[0146] In some optional embodiments, in the above Fig. 9 Based on the embodiment shown, Fig.10 As shown, the third processing module 53 may include:

[0147] The first processing unit 531 is used to determine the number of bad pixels and bad pixel location information according to the first bad pixel detection information.

[0148] The second processing unit 532 is used to determine bad pixel distribution information according to the bad pixel quantity and bad pixel location information.

[0149] The third processing unit 533 is used to determine the bad pixel concentration state according to the bad pixel distribution information.

[0150] The fourth processing unit 534 is used to determine the confidence level that the first sensing channel is in an aging state according to the bad pixel concentration state.

[0151] In some optional embodiments, the fourth processing unit 534 may be specifically configured to:

[0152] In response to the bad pixel concentration state being a first state, second bad pixel detection information of at least one second perception channel having an overlapping viewing angle with the first perception channel is determined; based on the second bad pixel detection information of at least one second perception channel, the bad pixel quantity and bad pixel position information corresponding to the first bad pixel detection information are verified to obtain a verification result; based on the bad pixel concentration state and the verification result, the confidence that the first perception channel is in an aging state is determined.

[0153] In some optional embodiments, the fourth processing unit 534 may be specifically configured to:

[0154] In response to the verification result satisfying the first condition, image data corresponding to the first original sensor data is determined; based on the image data, a quality score corresponding to the image data is determined; and according to the bad pixel concentration state, the verification result and the quality score, the confidence that the first perception path is in an aging state is determined.

[0155] In some optional embodiments, such as Fig.10 As shown, the device of the embodiment of the present disclosure may further include: a fifth processing module 61 and a sixth processing module 62 .

[0156] The fifth processing module 61 is used to determine the image data corresponding to the first original sensor data.

[0157] Among them, the fifth processing module 61 can be an image signal processing module in the first perception channel, or the fifth processing module 61 can obtain image data corresponding to the first original sensor data from the image signal processing module in the first perception channel.

[0158] The sixth processing module 62 is used to determine a quality score corresponding to the image data based on the image data.

[0159] The fourth processing unit 534 may be specifically configured to determine the confidence level that the first perception channel is in an aging state according to the bad pixel concentration state and the quality score.

[0160] In some optional embodiments, the sixth processing module 62 can be specifically used to: downsample the image data to obtain target image data of a preset size, where the preset size is smaller than the original size of the image data; and determine a quality score corresponding to the image data based on the target image data of the preset size.

[0161] In some optional embodiments, the image data may be downsampled by a hardware pyramid module in the perception system to improve downsampling efficiency. Specifically, the sixth processing module 62 may transmit the image data to the hardware pyramid module in the perception system, and downsample the image data by the hardware pyramid module to obtain downsampled target image data of a preset size.

[0162] Fig.11 It is a structural diagram of a monitoring device for a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0163] In some optional embodiments, in the above Fig. 9 Based on the embodiment shown, Fig.11 As shown, the third processing module 53 may include: a fifth processing unit 53a, used to determine the confidence that the first perception channel is in an aging state based on the first bad pixel detection information and historical bad pixel detection information of at least one historical moment of the first perception channel.

[0164] Fig.12 It is a structural diagram of a monitoring device for a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0165] In some optional embodiments, based on any of the above embodiments, Fig.12 As shown, the device of the embodiment of the present disclosure may also include:

[0166] The seventh processing module 55 is used to determine a target pixel area corresponding to an aging state in response to the monitoring result satisfying a preset condition.

[0167] The eighth processing module 56 is configured to determine image cropping information according to the target pixel area in response to the target pixel area being located within the first preset area range.

[0168] The sending module 57 is used to send the image cropping information to the perception system, so that the perception system crops the image according to the image cropping information, and obtains the perception result based on the cropped image.

[0169] In some optional embodiments, such as Fig.12 As shown, the apparatus of the embodiment of the present disclosure may further include: a ninth processing module 58 and a tenth processing module 59 .

[0170] The ninth processing module 58 is used to determine the driving state of the movable device where the first sensing path is located in response to the target pixel area being located in the second preset area range.

[0171] The tenth processing module 59 is used to execute processing instructions corresponding to the driving state.

[0172] In some optional embodiments, the tenth processing module 59 can be specifically used to: in response to the driving state being in motion, output a prompt message for suspending the use of the intelligent driving function; or, in response to the driving state being stationary, send aging notification information to the server corresponding to the movable device.

[0173] Fig.13 It is a structural diagram of a monitoring device for a sensing path provided by yet another exemplary embodiment of the present disclosure.

[0174] In some optional embodiments, based on any of the above embodiments, Fig.13 As shown, the apparatus of the embodiment of the present disclosure may further include: an acquisition module 71 and an update processing module 72 .

[0175] The acquisition module 71 is used to acquire an update data packet of the monitoring method for the sensing path from the server.

[0176] The update processing module 72 is used to update the monitoring method of the sensing path according to the update data packet, obtain the updated monitoring method of the sensing path, and use it to monitor the first sensing path.

[0177] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0178] Exemplary Electronic Devices

[0179] Fig.14 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0180] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0181] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0182] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0183] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (such as cameras, cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to sense the surrounding environment, that is, to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to realize the positioning of movable devices (such as self-vehicles, robots, etc.) where electronic devices are located. Positioning sensors may include, for example, inertial measurement units (Inertial Measurement Unit, referred to as: IMU), global positioning systems (Global Positioning System, referred to as: GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.

[0184] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.

[0185] Of course, to simplify, Fig.14 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0186] Exemplary computer program products and computer-readable storage media

[0187] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the methods of various embodiments of the present disclosure described in the above “Exemplary Methods” section.

[0188] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0189] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0190] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0191] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0192] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for monitoring a sensing path, comprising: Acquire first raw sensor data collected by a first image sensor in a first sensing path; Determining first bad pixel detection information based on the first original sensor data; Determining, based on the first bad pixel detection information, a confidence level that the first sensing path is in an aging state; Based on the confidence level, a monitoring result of the first perception pathway is determined.

2. The method according to claim 1, wherein: The step of determining the confidence level that the first sensing path is in an aging state based on the first bad pixel detection information includes: Determine the number of bad pixels and bad pixel location information according to the first bad pixel detection information; Determining bad pixel distribution information according to the bad pixel quantity and bad pixel location information; Determining a bad pixel concentration state according to the bad pixel distribution information; The confidence level that the first sensing path is in an aging state is determined according to the bad pixel concentration state.

3. The method according to claim 2, wherein: The step of determining the confidence level that the first sensing path is in an aging state according to the bad pixel concentration state includes: In response to the bad pixel concentration state being a first state, determining second bad pixel detection information of at least one second perception channel having an overlapping viewing angle with the first perception channel; Based on the second bad pixel detection information of the at least one second sensing channel, verifying the bad pixel quantity and bad pixel location information corresponding to the first bad pixel detection information to obtain a verification result; The confidence level that the first sensing path is in an aging state is determined according to the bad pixel concentration state and the verification result.

4. The method according to claim 3, wherein: The step of determining the confidence level that the first sensing path is in an aging state according to the bad pixel concentration state and the verification result includes: In response to the verification result satisfying a first condition, determining image data corresponding to the first raw sensor data; Based on the image data, determining a quality score corresponding to the image data; The confidence level that the first sensing path is in an aging state is determined according to the bad pixel concentration state, the verification result and the quality score.

5. The method according to claim 2, wherein: The method further comprises: determining image data corresponding to the first raw sensor data; Based on the image data, determining a quality score corresponding to the image data; The step of determining the confidence level that the first sensing path is in an aging state according to the bad pixel concentration state includes: The confidence level that the first perception channel is in an aging state is determined according to the bad pixel concentration state and the quality score.

6. The method according to claim 1, wherein: The step of determining the confidence level that the first sensing path is in an aging state based on the first bad pixel detection information includes: Based on the first bad pixel detection information and historical bad pixel detection information of at least one historical moment of the first perception channel, the confidence level that the first perception channel is in an aging state is determined.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: In response to the monitoring result satisfying a preset condition, determining a target pixel area corresponding to the aging state; In response to the target pixel area being located in a first preset area range, determining image cropping information according to the target pixel area; The image cropping information is sent to a perception system, so that the perception system crops the image according to the image cropping information, and obtains a perception result based on the cropped image.

8. The method according to claim 7, wherein: The method further comprises: In response to the target pixel area being located within a second preset area range, determining a driving state of the movable device where the first sensing path is located; Execute a processing instruction corresponding to the driving state.

9. The method according to claim 8, wherein: The executing a processing instruction corresponding to the driving state includes: In response to the driving state being motion, outputting a prompt message for suspending the use of the intelligent driving function; or, In response to the driving state being stationary, aging notification information is sent to a server corresponding to the movable device.

10. The method according to claim 9, wherein: The method further comprises: Obtaining an update data packet of a monitoring method for the sensing path from the server; The monitoring method of the sensing path is updated according to the update data packet to obtain an updated monitoring method of the sensing path, which is used to monitor the first sensing path.

11. A monitoring device for sensing a path, comprising: A first processing module, used to obtain first raw sensor data collected by a first image sensor in a first sensing path; A second processing module, configured to determine first bad pixel detection information based on the first original sensor data; A third processing module, configured to determine, based on the first bad pixel detection information, a confidence level that the first sensing path is in an aging state; The fourth processing module is used to determine the monitoring result of the first perception channel based on the confidence level.

12. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 10.

13. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-10.

Images