Fault testing method and system for unmanned autonomous system

Abstract

The invention discloses a fault testing method and system for an unmanned autonomous system, and belongs to the technical field of system testing. The method comprises the following steps: modeling a sensor fault, and establishing electrical connection between a sensor fault model and an unmanned autonomous system and a simulator; acquiring sensor data in the simulator, and processing the sensor data based on the sensor fault model to obtain fault sensor data; and enabling the data of the fault sensor to flow into the unmanned autonomous system so as to complete the test of the unmanned autonomous system in the fault mode. According to the invention, the operation safety and reliability of the unmanned autonomous system in a real scene can be improved.

Description

Technical Field

[0001] This invention belongs to the field of system testing technology, and in particular, it designs a fault testing method and system for unmanned autonomous systems. Background Technology

[0002] Unmanned autonomous systems rely on various sensors (such as cameras and lidar) to collect real-time information about the surrounding environment. This data is then fused and analyzed through a high-performance computing platform to generate decision-making commands to control system operation. However, in practical applications, unmanned autonomous systems face extremely complex environments and numerous challenges. Sensor malfunctions are diverse, including hardware damage (such as broken lenses or lasers), environmental interference (strong light, rain, and fog affecting imaging and point cloud quality), and human factors (collision damage). Variable weather severely impacts perception capabilities; for example, rain and snow blur camera images and increase lidar noise, fog and haze reduce visibility, and extreme temperatures affect sensor sensitivity. Differences in the anti-interference capabilities of different sensors can also lead to data fusion conflicts. Furthermore, over long-term use, sensors are prone to gradual failure due to mechanical vibration (loose lenses, lidar misalignment), thermal aging (material degradation, performance degradation), electrical faults (voltage fluctuations, electrostatic damage), and chemical corrosion (interface oxidation, component moisture). Initially, this manifests as performance degradation, but eventually, it can lead to complete functional loss. Even more challenging is that some damage is insidious, only causing sudden malfunctions under specific environmental conditions. Furthermore, factors such as sudden changes in lighting and electromagnetic interference on actual roads can also affect sensor reliability. It is evident that sensors are susceptible to a variety of factors, making it difficult to fully account for all the data corruption modes that can occur. These challenges stem not only from external environmental factors but also from the inherent reliability issues of the sensors themselves, all of which can pose serious threats to the safety and stability of unmanned autonomous systems.

[0003] Current research on constructing sensor fault models for unmanned autonomous systems primarily focuses on the failure scenarios of single-type sensor faults in extreme environments and the generation of faults from static scene data. It lacks comprehensive and systematic research on constructing sensor fault models and how to inject these faults into operating unmanned autonomous systems. Therefore, this invention delves into this aspect. When sensors malfunction, whether an unmanned autonomous system based on multi-sensor fusion can still make safe decisions and control actions is crucial for ensuring the operational safety and reliability of unmanned autonomous systems in real-world scenarios. How to construct sensor fault models and inject sensor faults into unmanned autonomous systems for testing remains an unsolved problem.

[0004] Existing research (see: Zhong Z, Hu Z, Guo S, et al. Detecting multi-sensor fusion errors in advanced driver assistance systems [C] / / proceedings of the 31st ACMSIGSOFT International Symposium on Software Testing and Analysis. 2022:493-505; or Gao X, Wang Z, Feng Y, et al. Multitest: Physical-aware object insertion for testing multisensor fusion perception systems[C] / / Proceedings of the IEEE / ACM46th International Conference on Software Engineering.2024:1-13; or Xiong Z, XuH, Li W, et al.Multi-source adversarial sample attack on autonomous vehicles[J].IEEE Transactions on Vehicular Technology, 2021, 70(3):2822-2835.) mainly focuses on building extreme cases of operating environments to detect errors in the perception model of unmanned autonomous systems, while ignoring the impact of perception errors on system-level safety (such as decision-making and action control).A few studies (Cao Y, Xiao C, Cyr B, et al. Adversarial sensor attack on lidar-based perception in autonomous driving[C] / / Proceedings of the 2019ACM SIGSAC conference on computer and communications security.2019:2267-2281; or Cao Y, Wang N, Xiao C, et al. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under physical-world attacks[C] / / 2021IEEE symposium on security and privacy(SP).IEEE,2021:176-194.) tested the impact of perception errors in unmanned autonomous systems on unmanned autonomous systems, but they generated adversarial sensor inputs from the scene rather than taking into account inherent sensor failures. Secci et al. (Secci F, Ceccarelli A. On failures of RGB cameras and their effects in autonomous driving applications[C] / / 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020: 13-24.) discovered safety violations in unmanned autonomous systems by injecting camera faults into them. This approach focuses only on the impact of camera faults on single-sensor (camera) unmanned autonomous systems, neglecting the role of lidar faults and multi-sensor fusion in the perception of unmanned autonomous system operation. A key advantage of multi-sensor fusion perception is its ability to compensate for errors of individual sensors; therefore, existing methods cannot inject sensor faults into industrial-grade autonomous unmanned systems based on multi-sensor fusion.In addition, in existing work (Gao driving applications[J].IEEETransactions on Dependable and Secure Computing, 2022, 20(4):2731-2745.) are all based on static KITTI datasets to generate fault images such as blurry and dust-occluded images to add sensor faults to the sensor data. However, they do not consider all fault modes. For example, for cameras, they do not consider the blurring caused by accumulated dust in the environment or the overexposure of local images caused by strong light interference. For lidar, they do not consider the failure of the laser emitter to emit laser beams due to wear and aging or the deflection and displacement of the laser emitter caused by bumps during vehicle movement. Furthermore, they do not consider the dynamic nature of sensor data input, do not inject faults into the running unmanned autonomous system, and do not consider injecting sensor faults into the operation of the complete unmanned autonomous system for testing. Summary of the Invention

[0005] This invention discloses a fault testing method and system for unmanned autonomous systems. By generating various types of sensor faults and implementing fault injection, it improves the safety and reliability of unmanned autonomous systems in real-world scenarios.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following:

[0007] A fault testing method for unmanned autonomous systems, the method comprising:

[0008] Model sensor faults and establish electrical connections between the sensor fault model and the unmanned autonomous system and simulator;

[0009] Acquire sensor data from the simulator and process the sensor data based on the sensor fault model to obtain faulty sensor data;

[0010] The fault sensor data is fed into the unmanned autonomous system to complete the test of the unmanned autonomous system in fault mode.

[0011] Furthermore, when the sensor type is a camera, the types of fault sensor data include: fault sensor data caused by deflection and offset, fault sensor data caused by internal dust accumulation, fault sensor data caused by lens breakage, fault sensor data caused by lens brightness changes, fault sensor data caused by image blur, fault sensor data caused by internal scattered points, fault sensor data caused by lens occlusion, fault sensor data caused by dust, fault sensor data caused by raindrops, fault sensor data caused by snow particles, fault sensor data caused by fog, fault sensor data caused by ice, fault sensor data caused by overexposure, and fault sensor data caused by white balance shift.

[0012] Furthermore, the fault sensor data P′=T generated by deflection and offset d P, Among them, t d Indicates the offset or misalignment between the camera and the axis, R d This indicates the angular misalignment of the camera around its axis, and P represents the original image;

[0013] Fault sensor data I caused by internal dust accumulation d =∫ Ω f t (w i ,w0;τ,σ a ,σ s ,g,n)·L i (w i )dw i Among them, ∫ Ω L represents the integral over the direction space Ω. i (w i ) is from the direction w i Radiance, f t (w i ,w0;τ,σ a ,σ s (g,n) represents the absorption coefficient τ and scattering coefficient σ. a Phase function parameter σ s Under the influence of refractive index g and the amount of dust accumulation n, from the direction w i The proportion of light rays scattered into the outgoing direction w0;

[0014] Fault sensor data I caused by lens breakage b=I e ×H psf +N; where H psf Let N represent a Gaussian function with a variable diffusion radius, and I represent additional interference. e Represents the pixel intensity of the original image;

[0015] Fault sensor data I generated by lens brightness changes L =α L ·I o (x,y,t)+β L ·I a (x,y,t)+γ L ·I g (x,y,t)+η·σ(t); where, I o (x,y,t) represents the pixel intensity of the original image, (x,y) represents the image pixel, and t represents time. a (x,y,t) represents the ambient light entering the lens and being magnified or attenuated, I g (x,y,t) represents internal glare, α(t) represents the time-varying random noise component, η represents the noise intensity coefficient, and α L β is the brightness attenuation factor. L γ is the ambient light scaling factor. L This is a scaling factor for glare intensity;

[0016] Faulty sensor data caused by image blur Where k is the kernel size, N(x,y) represents the Gaussian noise superimposed on the blurred image, and C(i,j) represents the weight of the Gaussian blur kernel corresponding to each pixel position;

[0017] Fault sensor data generated by internal scattering Where N s (x,y), N g (x,y), N b (x,y) is the random noise function for each color channel, representing color scattering noise, R s (x,y), G s (x,y), B s (x,y) represent the noise interference levels of the red, green, and blue channels, respectively, where R(x,y), G(x,y), and B(x,y) represent the original noise levels of the red, green, and blue channels, respectively. α s α g α b These represent the noise intensity coefficients for the red, green, and blue channels, respectively.

[0018] Fault sensor data I caused by lens obstruction u =(1-ψ(M))·Ie +ψ(M)·ψ(I l ); where ψ is an unsupervised Cycle-GAN that scales images based on occlusion, and the polygonal mask is... I l The image is a randomly occluded pattern, where m represents the degree of occlusion.

[0019] Fault sensor data caused by external contamination Where X represents the location of the event in the camera view. The set of positions, ∏ i∈X Multiply the Poisson distributions of each pixel position i in the location set X, and the Poisson distribution... k i λ represents the number of events occurring at pixel position i. i The rate parameter representing pixel position i during the Poisson process;

[0020] Dust-generated fault sensor data I du (x,y)=I e (x,y)×T(x,y)+S du (x,y); where T(x,y) is the transmission coefficient diagram, S du (x,y) represents the scattered light produced at pixel (x,y) by diffraction and scattering caused by dust particles;

[0021] Fault sensor data generated by raindrops I r (x,y)=(1-L r (x,y))·I e (x,y)+L r (x,y)·(t r ·I e (x,y)+(1-t r )·N r (x,y)); where N r (x,y) represents the random noise factor simulating refractive distortion, t r L is the transparency factor. r (x,y) represents the rain line mask;

[0022] Fault sensor data I generated by snow particles sw =(1-R) a,λ (x,y))·G(I e (x,y))+R a,λ (x,y)·N sw Among them, R a,λ I represents the spectral albedo at wavelength λ. sw Images captured by a camera with snow particle interference, N sw G(I) represents the scattered light caused by the reflection and refraction properties of snow particles.e (x,y) is used to simulate the light scattering effect caused by the translucent properties of snowflakes based on their size;

[0023] Fault sensor data I generated by fog m (x,y)=I e (x,y)·L(x,y)+T(x,y)·(1-L(x,y)); where L is the transmission function of light intensity attenuation caused by fog, and T is the ambient light scattering caused by fog;

[0024] Ice-generated fault sensor data I ce (x,y)=I e (x,y)·L c (x,y)+S ce (x,y)·(1-L c (x,y))+A(x,y); where, L c (x,y) represents the distortion at pixel (x,y) after light passes through the ice layer, S ce (x,y) represents the light refraction caused by the irregular ice surface, and A(x,y) represents the degree of light diffusion by the ice layer;

[0025] Faulty sensor data caused by overexposure Where f is the spatial frequency, S p (f) represents the signal power spectrum, N p (f) represents the noise power spectrum, and B is the Nyquist frequency;

[0026] Fault sensor data I caused by white balance offset f (x,y)=clip(I wb (x,y),0,255); where clip represents the pixel value used for initial white balance offset within the range of 0 to 255. s R s G s B T represents the color temperature adjustment coefficient for the red, green, and blue channels. wb Indicates the adjusted color temperature, I R I G I B These represent the pixel intensities of the red, green, and blue channels, respectively.

[0027] Furthermore, when the sensor type is lidar, the types of fault sensor data include one or more of the following: fault sensor data caused by deflection, fault sensor data caused by beam loss, fault sensor data caused by line fault, fault sensor data caused by electromagnetic interference, fault sensor data caused by crosstalk, fault sensor data caused by rain and snow pollution, and fault sensor data caused by strong light interference.

[0028] Furthermore, the fault sensor data P generated by the deflection j (x,y,z)=(ε,Z j ); where the mean of the Z-coordinate of the j-th grid is... n j Z represents the number of points in the j-th grid. ji The offset ε represents the Z-coordinate value of the i-th coordinate data contained in the j-th grid, and the offset ε = N1ε1 + N2ε2 + N3ε3, where N1, N2, and N3 represent the weight factors in the x, y, and z directions, respectively, and ε1, ε2, and ε3 represent the offsets in the x, y, and z directions, respectively.

[0029] Fault sensor data caused by beam loss Where, α bl β represents the density decay rate. bl The intensity decay rate is represented by P0, the original point cloud data is represented by d0, the initial beam density is represented by I0, and the decay time is represented by t.

[0030] Fault sensor data generated by line fault in, Let I represent the spatial coordinates of the i-th point at time t. i This indicates the signal strength value returned at the i-th point. Indicates a normal distribution. Let represent the variance of Gaussian noise in the spatial dimension, respectively. This represents the variance of noise in the intensity measurement;

[0031] Fault sensor data caused by electromagnetic interference Among them, -d A-E (t) represents the distance between the unmanned autonomous system and the electromagnetic source at time t, σ E (t) represents the standard deviation of electromagnetic interference of noise at time t;

[0032] Crosstalk-induced fault sensor data Where N(t) represents the number of interference points observed at time t, n ct Let λ represent the number of interference points, and λ represent the rate parameter of time in the Poisson process.

[0033] Faulty sensor data caused by rain and snow pollution Where R is the laser radar ray radius, ξ is the laser reflectivity, and α lf For the horizontal field of view, β lf For vertical field of view, Let C(α) be the extinction coefficient of the laser signal, exp represent the exponential function, and C(α) be the extinction coefficient of the laser signal. lf ,β lf This reflects the effect of the propagation angle on signal attenuation;

[0034] Fault sensor data caused by strong light interference Among them, single scattering return per unit area τ sl cτ is the pulse length. sl This indicates the spatial resolution of the lidar. Let σ be the aperture area, z be the distance inside the cloud, H be the distance to the cloud base, and σ be the distance to the cloud base. s Let σ be the cloud scattering coefficient. e P(π) is the extinction coefficient, and P(π) is the scattering phase function.

[0035] Furthermore, establish the electrical connection between the sensor fault model and the unmanned autonomous system and simulator, including:

[0036] Delete the sensor pointing to channel node C in the simulator. s The channel; wherein, the channel node C s Channel nodes used for acquiring sensor data in unmanned autonomous systems;

[0037] Construct channel node C′ s ;

[0038] The sensor in the simulator is pointed to channel node C′. s1 The channel and the channel node C′ s1 Channel pointing to the sensor model;

[0039] Construct a sensor model pointing to channel node C s The channel.

[0040] Further, sensor data from the simulator is acquired, and this sensor data is processed based on the sensor fault model to obtain faulty sensor data, including:

[0041] At channel node C′ s Monitor the data output by sensor S in the simulator, acquire the sensor data in real time, and store the sensor data in message queue Q;

[0042] Based on the selected fault sensor data model M tRetrieve the corresponding sensor data Data from message queue Q. s and will transfer sensor data. s Unpack the data to obtain sensor data in a specific data format;

[0043] Based on fault sensor data model M t The sensor data in this specific data format is processed to obtain fault sensor data.

[0044] Furthermore, the fault sensor data is fed into the unmanned autonomous system, including:

[0045] Store the fault sensor data into message queue Q′. s ;

[0046] Message queue Q′ s Send to channel node C s This is to enable the unmanned autonomous system to acquire the sensor data of the malfunction.

[0047] A fault testing system for unmanned autonomous systems, the system comprising:

[0048] The fault construction module is used to model sensor faults and establish the electrical connection between the sensor fault model and the unmanned autonomous system and simulator.

[0049] The data processing module is used to acquire sensor data from the simulator and process the sensor data based on the sensor fault model to obtain fault sensor data.

[0050] The data transmission module is used to transmit fault sensor data into the unmanned autonomous system in order to complete the testing of the unmanned autonomous system in fault mode.

[0051] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the fault testing method for unmanned autonomous systems described above.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects.

[0053] 1) Sensor faults are divided into active faults and passive faults. For active faults, the damage modes of sensor components are fully analyzed, and for passive faults, the environmental influencing factors are fully analyzed, enabling a systematic and comprehensive analysis of fault models.

[0054] 2) A mathematical model of sensor faults was constructed, and mathematical formulas were used to interpret and construct the sensor model, which can reasonably explain the rationality of the fault model;

[0055] 3) A fault injection technique is proposed, which involves constructing a new channel node and modifying the original sensor's channel node to point to the new channel node, thereby intercepting the original sensor data and injecting faults into it.

[0056] 4) By injecting fault models into the unmanned autonomous system and simulator operating environment, and by listening to sensor data to inject faults in real time and forwarding fault sensor data, faults can be injected into the sensor data in dynamic operation, thereby improving the safety and reliability of the unmanned autonomous system in real-world scenarios. Attached Figure Description

[0057] Figure 1 This is a flowchart of a fault testing method for unmanned autonomous systems.

[0058] Figure 2 For camera sensor components.

[0059] Figure 3 This is a lidar sensor assembly.

[0060] Figure 4 This is a schematic diagram of the fault.

[0061] Figure 5 This is a diagram illustrating a camera malfunction.

[0062] Figure 6 This is a schematic diagram of a lidar malfunction.

[0063] Figure 7 This is a schematic diagram showing the electrical connection between the sensor fault model and the unmanned autonomous system and simulator.

[0064] Figure 8 This is a diagram illustrating data monitoring.

[0065] Figure 9 A diagram illustrating fault injection.

[0066] Figure 10 Diagram illustrating fault data forwarding. Detailed Implementation

[0067] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0068] In the actual operating environment of unmanned autonomous systems, the sensors carried by these systems may encounter various problems affecting their accuracy and reliability. Extreme temperature conditions can also significantly impact sensor performance: low temperatures can cause fogging or frosting on sensor lenses, affecting light transmittance; high temperatures may increase thermal noise in image sensors, reducing image quality. Laser emitters in lidar may experience wavelength drift at high temperatures, reducing ranging accuracy; while the radio frequency chips in millimeter-wave radar may experience signal attenuation at extreme temperatures. Furthermore, the dark current of CMOS sensors changes significantly with temperature, especially in low-temperature environments where the signal-to-noise ratio drops sharply, affecting imaging quality under low-light conditions.

[0069] These problems can be mainly categorized into two types: active faults and passive faults. Active faults originate from internal sensor malfunctions or damage, usually caused by component failure or wear, directly leading to functional failure or performance degradation. Passive faults, on the other hand, are triggered by external environmental factors (such as obstacles in the environment or weather conditions), rather than damage or malfunction of the sensor itself. These external factors weaken the sensor's ability to accurately capture or interpret perceived data. To systematically simulate active and passive sensor faults in traffic scenarios, this method classifies them from two dimensions: sensor components and environmental factors. Based on this, a fault injection model is designed and implemented to simulate potential faults in cameras and lidar in unmanned autonomous systems under real-world operating conditions.

[0070] This method systematically models camera and lidar failures, simulating various failures that cameras and lidar may encounter and experience in real-world operating environments, including active failures caused by sensor component issues and passive failures caused by environmental interference. During the operation of the unmanned autonomous system and the simulator, sensor data from the simulator is intercepted and sensor failures are injected. This fault-injected sensor data is then fed back into the unmanned autonomous system, thus completing the sensor failure injection. This invention utilizes a failure model to inject failures into multi-sensor fusion unmanned systems, injecting both active and passive camera and lidar failures into the joint simulation of the unmanned autonomous system and the simulator, thereby improving the safety and reliability of the unmanned autonomous system operating in real-world scenarios.

[0071] Specifically, the fault testing method for unmanned autonomous systems of the present invention, such as... Figure 1 As shown, it includes the following steps.

[0072] Step 1: Model sensor faults and establish electrical connections between the sensor fault model and the unmanned autonomous system and simulator.

[0073] 1) Analysis of fault types.

[0074] This invention categorizes each type of sensor fault into active faults and passive faults. The sensor faults refer to perception errors that occur during the operation of the unmanned autonomous system. Active faults are caused by damage to the sensor's own components, while passive faults are caused by the sensor's own components not being damaged. The sensors may include cameras and lidar.

[0075] For cameras, the main consideration is to divide them into five components: lens, camera body, image sensor (CIS), image signal processor (ISP), and external connections, such as... Figure 2 As shown. Active faults of these five components of the camera are analyzed, and then environmental interference factors are mainly analyzed for passive faults.

[0076] For lidar, the main consideration is to divide it into four components: a protective housing, a receiver, a transmitter, and a receiver, such as... Figure 3 As shown, the active faults of the four components of the lidar are analyzed, and then the environmental interference factors are mainly analyzed for passive faults.

[0077] Based on the summary and analysis of the above active and passive faults of cameras and LiDAR, we can derive LiDAR fault tables and camera fault tables, as shown in Table 1.

[0078] Table 1 Fault Classification of Camera and LiDAR Sensors

[0079]

[0080]

[0081] 2) Fault construction.

[0082] For each type of sensor fault, this invention models and simulates the fault that would occur in a real operating environment based on the analyzed sensor fault. Specifically, this invention models the fault based on the type S of the sensor used. t From the set of fault modes, select the one that matches the sensor type S. t Corresponding fault mode F t Construct sensor fault model M set , indicating Figure 4 As shown. S t It can be a camera or a lidar sensor; not all fault modes can be injected into the sensors. For example, beam loss fault modes are not injectable into unmanned autonomous systems that do not include lidar. For each S... t The following infeasible failure modes are specified: S t = For cameras, the infeasible failure modes are those that include "beam loss," "electromagnetic interference," or "crosstalk mode"; S t= For LiDAR, the infeasible failure modes include those containing "lens breakage", "aperture / iris failure", "imaging failure", "lens obstruction", "white balance shift" or "lens icing".

[0083] After determining the failure mode, the present invention will construct a model for it. Specifically, if the sensor type S t =Camera, regarding active and passive camera malfunctions, camera malfunctions such as Figure 5 As shown, its construction process is as follows:

[0084] Deflection and offset: P′=T d P; Among them, t d Indicates the offset or misalignment between the camera and the axis, R d This represents the angular displacement of the camera about its axis, P′ represents the transformed image, P represents the original image, and T... d This indicates that the camera is misaligned with the axis.

[0085] Internal dust accumulation: I d =∫ Ω f t (w i ,w0;τ,σ a ,σ s ,g,n)·L i (w i )dw i , where I d Indicates the area covered by dirt, L i (w i ) is from the direction w i Radiance, f t (w i (w0) indicates from direction w i The proportion of light rays scattered towards the outgoing direction w0. τ, σ a σ s , g, and n represent the absorption coefficient, scattering coefficient, phase function parameter, refractive index, and dust accumulation amount, respectively. These parameters are all material properties of the pollutants. Ω This indicates that the integral is performed over the direction space Ω.

[0086] Lens shattered: I b =I e ×H psf +N, where I e This represents the original image captured by the camera with an undamaged lens, where N represents additional interference such as irregular debris, and H represents... psf This represents a Gaussian function with a variable diffusion radius.

[0087] Lens brightness change: I L =αL ·I o (x,y,t)+β L ·I a (x,y,t)+γ L ·I g (x,y,t)+η·σ(t), where I o (x,y,t) represents the original pixel intensity, I a (x,y,t) represents the ambient light entering the lens and being magnified or attenuated, I g (x,y,t) represents internal glare. σ(t) is the time-varying random noise component, and η represents the noise intensity coefficient. α L This is a brightness attenuation factor, determined by the degree of lens obstruction or aperture malfunction. β L γ is the ambient light scaling factor. L is a scaling factor for glare intensity; both depend on the severity of component failure. t represents time, and (x,y) represents the number of image pixels.

[0088] Image blur: Where k is the kernel size, (x,y) are the pixel coordinates relative to the kernel center. N(x,y) represents the Gaussian noise superimposed on the blurred image, and C(i,j) represents the Gaussian blur kernel, corresponding to the weights associated with each pixel position. e This is the original image.

[0089] Internal scatter points: Where N s (x,y), N g (x,y), N b (x,y) represents the random noise function for each color channel, indicating color scattering noise. R s (x,y), G s (x,y), B s (x,y) represent the noise interference levels of the red, green, and blue channels, respectively, where R(x,y), G(x,y), and B(x,y) represent the original noise levels of the red, green, and blue channels, respectively. α s α g α b These represent the noise intensity coefficients for the red, green, and blue channels, respectively.

[0090] Lens occlusion: I u =(1-ψ(M))·I e +ψ(M)·ψ(I l ), Where I e Let ψ be the original image, and ψ be an unsupervised Cycle-GAN that scales the image based on occlusion. M is the polygonal mask, and I is the vector vector. lThis is a randomly occluded image. n represents the degree of occlusion, and I... u This refers to an obstruction on the lens.

[0091] External stains: in Indicates in the region In this context, π represents the probability of the number of events observed through a Poisson process. i∈X Multiply the Poisson distributions of each pixel i in region X. Pois(k) i |λ i ) represents the form of the Poisson distribution, i represents the position of the i-th pixel, and λ is the rate parameter of the Poisson process. i X represents the number of events occurring at position i, and X represents the number of events occurring at the camera position. The set of locations.

[0092] Dust: I du (x,y)=I e (x,y)×T(x,y)+S du (x,y), where I e (x,y) represents the value at pixel (x,y) of the original image captured by a dust-free camera lens. T(x,y) is a transmittance map, representing the light attenuation at pixel (x,y) caused by dust coverage, with values ​​ranging from 0 (complete occlusion) to 1 (no attenuation). S du (x,y) represents the scattered light produced at pixel (x,y) by diffraction and scattering caused by dust particles, which introduces Gaussian blur and brightness distortion around the dust particles.

[0093] Raindrop: I r (x,y)=(1-L r (x,y))·I e (x,y)+L r (x,y)·(t r ·I e (x,y)+(1-t r )·N r (x,y)), where I e (x,y) represents the raw pixel intensity of the image captured by the camera when there is no raindrop interference, N r (x,y) represents the random noise factor simulating refractive distortion, t r L is a transparency factor used to simulate the partial occlusion effect of rain lines on the camera's field of view. r (x,y) represents the rain line mask, which describes the spatial distribution characteristics of rain lines on the lens surface.

[0094] Snowflakes: I sw =(1-R) a,λ (x,y))·G(Ie (x,y))+R a,λ (x,y)·N sw , where R a,λ This represents the spectral albedo at wavelength λ. e (x,y) represents the original image captured by the camera without snow particle interference, I sw Images captured by a camera with snow particle interference, N sw G(I) represents the noise term, indicating the scattered light caused by the reflection and refraction properties of snow particles. e (x,y) is a Gaussian intensity term used to simulate the light scattering effect caused by the translucent properties of snow particles based on their size.

[0095] Fog: I m (x,y)=I e (x,y)·L(x,y)+T(x,y)·(1-L(x,y)), where I e (x,y) represents the original pixel intensity at (x,y), L is the transmission function considering the light intensity attenuation caused by fog, and T is the ambient light scattering caused by fog, which is proportional to the intensity of the surrounding light source.

[0096] Ice: I ce (x,y)=I e (x,y)·L c (x,y)+S ce (x,y)·(1-L c (x,y))+A(x,y), where L c (x,y) represents the distortion at pixel (x,y) after light passes through the ice layer, S ce (x,y) represents the light refraction caused by the irregular ice surface. A(x,y) is the additional blur caused by light scattering, representing the degree of light diffusion by the ice layer.

[0097] Overexposure: Where f is the spatial frequency, S p (f) represents the signal power spectrum, N p (f) represents the noise power spectrum, and B represents the Nyquist frequency.

[0098] White balance shift: I f (x,y)=clip(I wb (x,y),0,255), Where I f (x, y) represents the pixel value after color change, and clip represents the range of pixel values, ensuring that each value is within the range of 0 to 255. wb (x, y) represents the pixel values ​​of the initial white balance shift. R s G sB T represents the color temperature adjustment coefficient for the red, green, and blue channels. wb Indicates the adjusted color temperature, I R I G I B These represent the red channel, green channel, and blue channel, respectively.

[0099] If sensor type S t =LiDAR, regarding active and passive faults of LiDAR, LiDAR faults are as follows Figure 6 As shown, the specific construction of the lidar fault model is as follows:

[0100] Offset: P j (x,y,z)=(ε,Z j ), ε=N1ε1+N2ε2+N3ε3, where P j (x,y,z) represents the value after the j-th grid offset, n j Z represents the number of points in the j-th grid. ji Z represents the Z-coordinate value of the i-th coordinate data contained in the j-th grid. j This represents the mean Z-coordinate of the j-th grid. ε1, ε2, and ε3 represent the offsets in the x, y, and z directions, respectively. N1, N2, and N3 represent the weighting factors in each direction, used to adjust for the influence of errors.

[0101] Beam loss: Where α bl β represents the density decay rate. bl This represents the intensity attenuation rate. P0 represents the raw point cloud data without injected faults, D0 is the initial beam density, I0 is the initial beam intensity, t represents the attenuation time, and P... bl This represents the point cloud data after beam loss.

[0102] Line fault: in Let I represent the spatial coordinates of the i-th point at time t. i This represents the signal strength value returned at the i-th point. Indicates a normal distribution. These represent the variances of Gaussian noise in the spatial dimension, reflecting the degree of distortion in distance and position measurements. It represents the variance of noise in the intensity measurement, reflecting the error in the received signal strength. This represents the data of point i after a line fault.

[0103] Electromagnetic interference: Where P oThis represents the raw point cloud data of the lidar when it is not subject to electromagnetic interference. -d A-E (t) represents the distance between the unmanned autonomous system and the electromagnetic source at time t. Represents a normal distribution, σ E (t) represents the standard deviation of electromagnetic interference at time t, and its value increases as the distance from the electromagnetic source decreases.

[0104] Crosstalk: Where N(t) represents the number of interference points (shot noise points) observed at time t, λ is the rate parameter of the Poisson process, representing the average number of interference points generated per unit time or unit distance, and n ct This represents the number of interference points.

[0105] Rain and snow pollution: Wherein, C(α) lf ,β lf This reflects the effect of the propagation angle on signal attenuation. P o This is the original point cloud of the lidar without injected faults, where R is the lidar ray radius, ξ is the laser reflectivity, and α... lf β is the horizontal field of view, and β is the vertical field of view. Let C(α) be the extinction coefficient of the laser signal, and exp represent the exponential function. lf ,β lf This reflects the effect of the propagation angle on signal attenuation. lf This represents point cloud data after rain and snow pollution.

[0106] Strong light interference: Where τ sl cτ is the pulse length. sl χ represents the spatial resolution of the lidar. s For a single scattering return per unit area, Let σ be the aperture area, z be the distance inside the cloud, H be the distance to the cloud base, and σ be the distance to the cloud base. s Let σ be the cloud scattering coefficient. e P(π) is the extinction coefficient, P(π) is the scattering phase function, and π represents the constant pi.

[0107] By constructing the above faults, a suitable sensor S will be obtained. t Fault mode model set M set .

[0108] 3) Electrical connection between sensor fault model and unmanned autonomous system and simulator.

[0109] For normal closed-loop simulation of unmanned autonomous systems, the data in the simulator... sIt requires sensor S to acquire the data, and then the data is directly input to the corresponding unmanned autonomous system's data channel node C via the sensor. s In this step, the present invention will construct a new channel node C′. s Modify the pointer to the original channel node C. s The channel is directed to the new node C′. s ,like Figure 7 and Figure 8 As shown. And simultaneously, this new channel node C′ s Monitor the data output by sensor S in the simulator and acquire the sensor data in real time, then feed it into the newly constructed channel node C′. s The acquired sensor data (Data) is stored using a message queue (Q). s .

[0110] Specifically, the original channel nodes corresponding to the cameras and lidar sensors of the unmanned autonomous system were C and C, respectively. camera and C lidar When sensor S t =The camera will build a connection with C camera The corresponding new node C′ camera The original camera channel C=C in the simulator camera This invention modifies it to C = C′ camera , and in C′ camera It will monitor sensor data in the simulator in real time. camera And store it in the corresponding message queue Q. camera In the middle; when sensor S t =LiDAR will build a network with C lidar The corresponding new node C′ lidar The original LiDAR channel C=C in the simulator lidar This invention modifies it to C = C′ lidar , and in C′ lidar It will monitor sensor data in the simulator in real time. lidar And store it in the corresponding message queue Q. lidar middle.

[0111] Through the above process, the present invention will be able to monitor sensor data in real time and store it for later use.

[0112] Step 2: Obtain sensor data from the simulator and process the sensor data based on the sensor fault model to obtain fault sensor data.

[0113] This invention selects and constructs sensor fault injection from a new channel node C′ sData obtained from s Thus, sensing data with sensor faults is obtained. s First, this invention will be based on fault mode F. t The set of failure mode models M obtained from failure construction set Select the corresponding fault model M t Then from the corresponding message queue Q s Extract the required sensor data. s and will transfer sensor data. s The data format d that can be processed by this invention is obtained by unpacking. s Secondly, the sensor fault model M t Add to data d s It is then packaged into fault data (Data'). s Finally, the fault data Data′ s Store in the corresponding fault message queue Q′ s The fault injection process is as follows: Figure 9 As shown.

[0114] Specifically, the newly constructed channel nodes are C′ camera and C′ lidar When channel node C = C′ camera First, it will be based on the camera's fault mode F camera The set of failure mode models M obtained from the failure construction module set Select the corresponding camera fault model M camrea Then from the corresponding camera message queue Q camera Extract the required sensor data. camera and will transfer sensor data. camera The data format d that can be processed by this invention is obtained by unpacking. camera Secondly, the camera sensor fault model M camera Add to data d camera It was then packaged as camera fault data. camera Finally, the fault data Data′ camera Store in the corresponding fault message queue Q′ camera When channel node C = C′ lidar First, it will be based on the lidar failure mode F lidar The set of failure mode models M obtained from the failure construction module set Select the corresponding lidar fault model M lidar Then from the corresponding lidar message queue Q lidar Extract the required sensor data. lidar and will transfer sensor data.lidar The data format d that can be processed by this invention is obtained by unpacking. lidar Secondly, the fault model M of the lidar sensor is... lidar Add to data d lidar It was then packaged as LiDAR fault data. lidar Finally, the fault data Data′ lidar Store in the corresponding fault message queue Q′ lidar .

[0115] Through the above process, the present invention will be able to inject fault data into the corresponding sensor data to obtain fault sensor data Data′. s And it can be stored for later use.

[0116] Step 3: Flow fault sensor data into the unmanned autonomous system to complete the test of the unmanned autonomous system in fault mode.

[0117] This invention will incorporate sensor data (Data′) after a fault. s Send the unmanned autonomous system to the new node C′ s Sensor data with sensor malfunctions flows into the corresponding sensor channel node C of the unmanned autonomous system. s First, this invention constructs a [structure / system] from C′. s To C s Data Channel s So that C′ s The data in C can be successfully transferred to C. s Then, the fault message queue Q′ obtained by the fault injection module is processed. s Data from fault sensor in the middle s Send to channel node C s In the middle, the unmanned autonomous system will automatically start from channel node C s Data is read from the system to enable the operation of the unmanned autonomous system. The fault data forwarding process is as follows: Figure 10 As shown.

[0118] Specifically, for the newly constructed channel node C′ camera and C′ lidar When channel node C = C′ camera Build a C' camera To C camera Data Channel camera Channel camera It will retrieve the fault message queue Q′ in real time. camera Data′ for obtaining camera fault data camera And forwarded to channel node C camera In the middle; when channel node C = C′lidar Build a C' lidar To C lidar Data Channel lidar Channel lidar It will retrieve the fault message queue Q′ in real time. lidar Data′ for obtaining camera fault data lidar And forwarded to channel node C lidar middle.

[0119] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A fault testing method for unmanned autonomous systems, characterized in that, The method includes: Model sensor faults and establish electrical connections between the sensor fault model and the unmanned autonomous system and simulator; Acquire sensor data from the simulator and process the sensor data based on the sensor fault model to obtain faulty sensor data; The fault sensor data is fed into the unmanned autonomous system to complete the test of the unmanned autonomous system in fault mode.

2. The method according to claim 1, characterized in that, When the sensor type is a camera, the types of fault sensor data include: fault sensor data caused by deflection and offset, fault sensor data caused by internal dust accumulation, fault sensor data caused by lens breakage, fault sensor data caused by lens brightness changes, fault sensor data caused by image blur, fault sensor data caused by internal scattered points, fault sensor data caused by lens occlusion, fault sensor data caused by dust, fault sensor data caused by raindrops, fault sensor data caused by snow particles, fault sensor data caused by fog, fault sensor data caused by ice, fault sensor data caused by overexposure, and fault sensor data caused by white balance shift.

3. The method according to claim 2, characterized in that, Fault sensor data P′=T generated by deflection and offset d P, Among them, t d Indicates the offset or misalignment between the camera and the axis, R d This indicates the angular misalignment of the camera around its axis, and P represents the original image; Fault sensor data I caused by internal dust accumulation d =∫ Ω f t (w i ,w0;τ,σ a ,σ s ,g,n)·L i (w i )dw i Among them, ∫ Ω L represents the integral over the direction space Ω. i (w i ) is from the direction w i Radiance, f t (w i ,w0;τ,σ a ,σ s (g,n) represents the absorption coefficient τ and scattering coefficient σ. a Phase function parameter σ s Under the influence of refractive index g and the amount of dust accumulation n, from the direction w i The proportion of light rays scattered into the outgoing direction w0; Fault sensor data I caused by lens breakage b =I e ×H psf +N; where H psf Let N represent a Gaussian function with a variable diffusion radius, and I represent additional interference. e Represents the pixel intensity of the original image; Fault sensor data I generated by lens brightness changes L =α L ·I o (x,y,t)+β L ·I a (x,y,t)+γ L ·I g (x,y,t)+η·σ(t); where, I o (x,y,t) represents the pixel intensity of the original image, (x,y) represents the image pixel, and t represents time. a (x,y,t) represents the ambient light entering the lens and being magnified or attenuated, I g (x,y,t) represents internal glare, α(t) represents the time-varying random noise component, η represents the noise intensity coefficient, and α L β is the brightness attenuation factor. L γ is the ambient light scaling factor. L This is a scaling factor for glare intensity; Faulty sensor data caused by image blur Where k is the kernel size, N(x,y) represents the Gaussian noise superimposed on the blurred image, and C(i,j) represents the weight of the Gaussian blur kernel corresponding to each pixel position; Fault sensor data generated by internal scattering Where N s (x,y), N g (x,y), N b (x,y) is the random noise function for each color channel, representing color scattering noise, R s (x,y), G s (x,y), B s (x,y) represent the noise interference levels of the red, green, and blue channels, respectively, where R(x,y), G(x,y), and B(x,y) represent the original noise levels of the red, green, and blue channels, respectively. α s α g α b These represent the noise intensity coefficients for the red, green, and blue channels, respectively. Fault sensor data I caused by lens obstruction u =(1-ψ(M))·I e +ψ(M)·ψ(I l ); where ψ is an unsupervised Cycle-GAN that scales images based on occlusion, and the polygonal mask is... I l The image is a randomly occluded pattern, where m represents the degree of occlusion. Fault sensor data caused by external contamination Where X represents the location of the event in the camera view. The set of positions, ∏ i∈X Multiply the Poisson distributions of each pixel position i in the location set X, and the Poisson distribution... k i λ represents the number of events occurring at pixel position i. i The rate parameter representing pixel position i during the Poisson process; Dust-generated fault sensor data I du (x,y)=I e (x,y)×T(x,y)+S du (x,y); where T(x,y) is the transmission coefficient diagram, S du (x,y) represents the scattered light produced at pixel (x,y) by diffraction and scattering caused by dust particles; Fault sensor data generated by raindrops I r (x,y)=(1-L r (x,y))·I e (x,y)+L r (x,y)·(t r ·I e (x,y)+(1-t r )·N r (x,y)); where N r (x,y) represents the random noise factor simulating refractive distortion, t r L is the transparency factor. r (x,y) represents the rain line mask; Fault sensor data I generated by snow particles sw =(1-R) a,λ (x,y))·G(I e (x,y))+R a,λ (x,y)·N sw Among them, R a,λ I represents the spectral albedo at wavelength λ. sw Images captured by a camera with snow particle interference, N sw G(I) represents the scattered light caused by the reflection and refraction properties of snow particles. e (x,y) is used to simulate the light scattering effect caused by the translucent properties of snowflakes based on their size; Fault sensor data I generated by fog m (x,y)=I e (x,y)·L(x,y)+T(x,y)·(1-L(x,y)); where L is the transmission function of light intensity attenuation caused by fog, and T is the ambient light scattering caused by fog; Ice-generated fault sensor data I ce (x,y)=I e (x,y)·L c (x,y)+S ce (x,y)·(1-L c (x,y))+A(x,y); where, L c (x,y) represents the distortion at pixel (x,y) after light passes through the ice layer, S ce (x,y) represents the light refraction caused by the irregular ice surface, and A(x,y) represents the degree of light diffusion by the ice layer; Faulty sensor data caused by overexposure Where f is the spatial frequency, S p (f) represents the signal power spectrum, N p (f) represents the noise power spectrum, and B is the Nyquist frequency; Fault sensor data I caused by white balance offset f (x,y)=clip(I wb (x,y),0,255); where clip represents the pixel value used for initial white balance offset within the range of 0 to 255. s R s G s B T represents the color temperature adjustment coefficient for the red, green, and blue channels. wb Indicates the adjusted color temperature, I R I G I B These represent the pixel intensities of the red, green, and blue channels, respectively.

4. The method according to claim 1, characterized in that, When the sensor type is lidar, the types of fault sensor data include one or more of the following: fault sensor data caused by deflection, fault sensor data caused by beam loss, fault sensor data caused by line fault, fault sensor data caused by electromagnetic interference, fault sensor data caused by crosstalk, fault sensor data caused by rain and snow pollution, and fault sensor data caused by strong light interference.

5. The method according to claim 4, characterized in that, Fault sensor data P generated by deflection j (x,y,z)=(ε,Z j ); where the mean Z-coordinate of the j-th grid is Zj. j = n j Z represents the number of points in the j-th grid. ji The offset ε represents the Z-coordinate value of the i-th coordinate data contained in the j-th grid, and the offset ε = N1ε1 + N2ε2 + N3ε3, where N1, N2, and N3 represent the weight factors in the x, y, and z directions, respectively, and ε1, ε2, and ε3 represent the offsets in the x, y, and z directions, respectively. Fault sensor data caused by beam loss Where, α bl β represents the density decay rate. bl The intensity decay rate is represented by P0, the original point cloud data is represented by D0, the initial beam density is represented by I0, and the decay time is represented by t. Fault sensor data generated by line fault in, Let I represent the spatial coordinates of the i-th point at time t. i This indicates the signal strength value returned at the i-th point. Indicates a normal distribution. Let represent the variance of Gaussian noise in the spatial dimension, respectively. This represents the variance of noise in the intensity measurement; Fault sensor data caused by electromagnetic interference Among them, -d A-E (t) represents the distance between the unmanned autonomous system and the electromagnetic source at time t, σ E (t) represents the standard deviation of electromagnetic interference of noise at time t; Crosstalk-induced fault sensor data Where N(t) represents the number of interference points observed at time t, n ct Let λ represent the number of interference points, and λ represent the rate parameter of time in the Poisson process. Faulty sensor data caused by rain and snow pollution Where R is the laser radar ray radius, ξ is the laser reflectivity, and α lf For the horizontal field of view, β lf For vertical field of view, Let C(α) be the extinction coefficient of the laser signal, exp represent the exponential function, and C(α) be the extinction coefficient of the laser signal. lf ,β lf This reflects the effect of the propagation angle on signal attenuation; Fault sensor data caused by strong light interference Among them, single scattering return per unit area τ sl cτ is the pulse length. sl This indicates the spatial resolution of the lidar. Let σ be the aperture area, z be the distance inside the cloud, H be the distance to the cloud base, and σ be the distance to the cloud base. s Let σ be the cloud scattering coefficient. e P(π) is the extinction coefficient, and P(π) is the scattering phase function.

6. The method according to claim 1, characterized in that, Establish electrical connections between sensor fault models and unmanned autonomous systems and simulators, including: Delete the sensor pointing to channel node C in the simulator. s The channel; wherein, the channel node C s Channel nodes used for acquiring sensor data in unmanned autonomous systems; Construct channel node C′ s ; The sensor in the simulator is pointed to channel node C′. s1 The channel and the channel node C′ s1 Channel pointing to the sensor model; Construct a sensor model pointing to channel node C s The channel.

7. The method according to claim 6, characterized in that, Acquire sensor data from the simulator, and process the sensor data based on the sensor fault model to obtain faulty sensor data, including: At channel node C′ s Monitor the data output by sensor S in the simulator, acquire the sensor data in real time, and store the sensor data in message queue Q; Based on the selected fault sensor data model M t Retrieve the corresponding sensor data Data from message queue Q. s and will transfer sensor data. s Unpack the data to obtain sensor data in a specific data format; Based on fault sensor data model M t The sensor data in this specific data format is processed to obtain fault sensor data.

8. The method according to claim 6, characterized in that, To allow fault sensor data to flow into the unmanned autonomous system, including: Store the fault sensor data into message queue Q′. s ; Message queue Q′ s Send to channel node C s This is to enable the unmanned autonomous system to acquire the sensor data of the malfunction.

9. A fault testing system for unmanned autonomous systems, characterized in that, The system includes: The fault construction module is used to model sensor faults and establish the electrical connection between the sensor fault model and the unmanned autonomous system and simulator. The data processing module is used to acquire sensor data from the simulator and process the sensor data based on the sensor fault model to obtain fault sensor data. The data transmission module is used to transmit fault sensor data into the unmanned autonomous system in order to complete the testing of the unmanned autonomous system in fault mode.

10. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the fault testing method for unmanned autonomous systems as described in any one of claims 1-8.

Images