Edge fatigue detection method and device based on adaptive threshold adjustment and medium

Abstract

The application relates to the technical field of intelligent detection, in particular to an edge fatigue detection method and device based on adaptive threshold adjustment and a medium. The method comprises the following steps: acquiring facial image data and facial infrared data of a real-time state of a driver; pre-processing the facial image data and the facial infrared data; positioning facial landmark points of the pre-processed facial image data and facial infrared data through a facial feature detection model to obtain an enhanced 3D landmark point image; comparing the enhanced 3D landmark point image with a physiological feature database to obtain physiological data; and calculating the physiological data through a dynamic threshold decision model and a multi-index fusion engine to obtain a fatigue decision. The method can improve environmental adaptability, realize individualized accurate adaptation, accurately adapt to individual differences and enhance multi-working-condition robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent detection, in particular to an edge fatigue detection method and device based on adaptive threshold adjustment and a medium. BACKGROUND

[0002] With the rapid development of highway traffic, fatigue driving has become one of the main causes of traffic accidents, which seriously threatens public traffic safety. In the prior art, the MindWave device of the NeuroSky company in the United States judges the fatigue degree through bioelectric signals such as electrocardiogram and electroencephalogram, but such a device needs to wear a special electrode device, which affects the driving comfort, and the signal distortion is easily caused by excessive sweating or motion artifacts, and the electromagnetic interference under complex working conditions cannot be coped with. The Driver Alert Control system of Volvo is to infer the fatigue state indirectly through vehicle parameters such as steering wheel angle and lane deviation distance, but the false positive rate of this system for experienced drivers is high, and the sensitivity decreases by 42% in scenes such as curve driving or traffic jams. Due to the need for multi-sensor fusion, the system complexity increases by more than 3 times. The current mainstream scheme is to use the OpenCV+Dlib framework to realize fatigue evaluation through facial feature point detection, but this framework relies on 68-point facial landmark detection, judges the PERCLOS index through a fixed threshold, and the processing speed of a single-core CPU is ≤20fps, and the accuracy decreases to 63% under light changes.

[0003] In summary, the prior art has the technical problems of insufficient environmental adaptability, poor individual difference adaptability, limited edge computing resources, and lack of multi-working-condition robustness. The technical problems existing in the related art need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose an edge fatigue detection method, device and medium based on adaptive threshold adjustment, which can improve environmental adaptability, realize individualized precise adaptation, accurately adapt to individual differences, and enhance multi-working-condition robustness.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application proposes an edge fatigue detection method based on adaptive threshold adjustment, which comprises:

[0006] obtaining facial image data and facial infrared data of a real-time state of a driver;

[0007] preprocessing the facial image data and the facial infrared data;

[0008] locating facial landmark points of the preprocessed facial image data and the facial infrared data through a facial feature detection model to obtain an enhanced 3D landmark point image;

[0009] The enhanced 3D landmark image is compared with a physiological feature database to obtain physiological data.

[0010] The physiological data is calculated by a dynamic threshold decision model and a multi-index fusion engine to obtain a fatigue decision.

[0011] In some embodiments, the face feature detection model includes a contrast-limited adaptive histogram equalization model and a MediaPipe face detection model.

[0012] In some embodiments, the data processing process of the contrast-limited adaptive histogram equalization model includes the following steps:

[0013] The preprocessed face image data and the face infrared data are divided into a plurality of non-overlapping small blocks;

[0014] The pixel gray value in each small block is calculated to obtain a histogram of each small block;

[0015] The histogram is cut according to a pixel threshold to obtain a contrast-limited histogram;

[0016] The contrast-limited histogram after cutting is normalized to calculate the cumulative distribution function of each gray level, and a new gray mapping relationship is obtained;

[0017] Adjacent small blocks are smoothed by overlapping block or fusing information of the contrast-limited histogram of adjacent regions in the interpolation process;

[0018] The pixel gray value in each small block is mapped back to the original image coordinate system through the new gray mapping relationship to obtain an enhanced image.

[0019] In some embodiments, the data processing process of the MediaPipe face detection model includes the following steps:

[0020] Multi-scale feature extraction of the enhanced image is performed by a lightweight convolutional neural network to obtain a real-time feature map;

[0021] The real-time feature map is accelerated according to heterogeneous acceleration hardware;

[0022] The real-time feature map is predicted by a graph convolutional network to obtain an enhanced 3D landmark image.

[0023] In some embodiments, the construction process of the physiological feature database includes the following steps:

[0024] Performing probability density estimation on the historical enhanced 3D landmark point image data obtained by processing the historical data through a Gaussian mixture model to obtain basic Gaussian mixture model parameters;

[0025] Performing incremental training on the Gaussian mixture model by continuously inputting new historical enhanced 3D landmark point image data, updating the parameters of the Gaussian mixture model until the confidence reaches a threshold value;

[0026] When the confidence reaches the threshold value, calculating the normal physiological range boundary based on the Gaussian mixture model to obtain a physiological feature threshold table;

[0027] Performing feature fusion on the physiological feature threshold table and the auxiliary physiological information through correlation analysis to obtain a multi-dimensional physiological feature vector;

[0028] Storing the multi-dimensional physiological feature vector according to a space-time index structure to obtain the physiological feature database.

[0029] In some embodiments, it further includes data validity verification, including the following steps:

[0030] Obtaining the historical enhanced 3D landmark point image data;

[0031] Performing non-fatigue verification on the physiological data through fatigue probability and multi-dimensional cross verification;

[0032] When the verification is passed, updating the physiological data to the physiological feature database;

[0033] When the verification is not passed, triggering abnormal processing, prolonging the collection time of obtaining the physiological data and repeating the non-fatigue verification, and if it is still not passed, sending a review request to the customer service end.

[0034] In some embodiments, the dynamic threshold decision model includes a double-layer dynamic threshold and Kalman filter dynamic adjustment; the physiological data is calculated through the dynamic threshold decision model and the multi-index fusion to obtain a fatigue decision, including the following steps:

[0035] Comparing the physiological data with the double-layer dynamic threshold to obtain a first fatigue judgment;

[0036] Performing smoothing processing on the first fatigue judgment through the Kalman filter dynamic adjustment to obtain a second fatigue judgment;

[0037] Preprocessing the second fatigue judgment;

[0038] Performing "or" logical combination and dynamic weight adjustment on the preprocessed second fatigue judgment to obtain the fatigue decision.

[0039] In some embodiments, an edge computing optimization model is further included, which comprises a lightweight neural network architecture, 4-bit hybrid quantization, spatial dynamic ROI focusing, and a parallel processing pipeline.

[0040] The lightweight neural network architecture is used for volume compression of each model.

[0041] The 4-bit hybrid quantization is used to reduce memory occupation and computation.

[0042] The spatial dynamic ROI focusing is used to dynamically adjust the detection region according to the face position.

[0043] The parallel processing pipeline is used to implement multi-level task overlap execution.

[0044] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes an edge fatigue detection device based on adaptive threshold adjustment, which comprises:

[0045] A data acquisition module is configured to acquire facial image data and facial infrared data of a real-time state of a driver.

[0046] A data processing module is configured to pre-process the facial image data and the facial infrared data, locate facial landmark points of the pre-processed facial image data and the facial infrared data through a facial feature detection model, and obtain an enhanced 3D landmark point image.

[0047] A data decision module is configured to compare the enhanced 3D landmark point image with a physiological feature database to obtain physiological data, and calculate the physiological data through a dynamic threshold decision model and a multi-index fusion engine to obtain a fatigue decision.

[0048] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement an edge fatigue detection method based on adaptive threshold adjustment.

[0049] The embodiments of the present application at least have the following beneficial effects: The present application provides an edge fatigue detection method, device and medium based on adaptive threshold adjustment, which obtains facial image data and facial infrared data of a real-time state of a driver, locates facial landmark points of preprocessed data through a facial feature detection model, obtains an enhanced 3D landmark point image, and greatly improves the environmental adaptability of the obtained data through the facial feature detection model, solves the detection failure problem of the traditional method in the extreme light, head posture change or occlusion scene. Finally, through a dynamic threshold decision model and a multi-index fusion engine, data obtained by comparing the enhanced 3D landmark point image with a physiological feature database is calculated to obtain a fatigue decision. The dynamic threshold decision model realizes individual difference precision adaptation, eliminates the false detection problem caused by the fixed threshold, and adapts to the physiological feature difference of the driver. The multi-index fusion engine enhances the robustness of the present application in the multi-working-condition environment, and solves the false alarm and missing alarm problems in the complex driving scene. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 FIG. 1 is a flowchart of an edge fatigue detection method based on adaptive threshold adjustment provided by the embodiments of the present application;

[0051] Figure 2 FIG. 4 is a flowchart of data processing of a contrast-limited adaptive histogram equalization model;

[0052] Figure 3 FIG. 5 is an algorithm schematic diagram of a contrast-limited adaptive histogram equalization model enhancement process;

[0053] Figure 4 FIG. 6 is a flowchart of MediaPipe face detection model data processing;

[0054] Figure 5 FIG. 7 is a flowchart of a physiological feature database construction process;

[0055] Figure 6 FIG. 8 is a schematic diagram of a physiological feature threshold table source program code;

[0056] Figure 7 FIG. 9 is a schematic diagram of a fatigue probability source program code;

[0057] Figure 8 FIG. 10 is a structural schematic diagram of an edge fatigue detection device based on adaptive threshold adjustment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of devices and methods consistent with some aspects of embodiments of the present application.

[0059] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0060] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0063] ECR: Eye Closure Rate, an eye movement related index, measures the proportion of eye closure frequency or duration, reflects the state of eye muscle relaxation, abnormal blinking, etc. when tired, such as when the duration of long-time continuous eye closure exceeds the standard.

[0064] MOR: Mouth Open Rate, a mouth related index, monitors the mouth opening frequency or mouth opening duration, and when tired, yawning, jaw relaxation, etc. can cause abnormal mouth state.

[0065] HNFR: Head Nodding / Frequency Rate, tracks head nodding frequency, tilt angle, or shaking amplitude, head control ability decreases when tired, frequent nodding, drooping, etc.

[0066] CLAHE: Contrast Limited Adaptive Histogram Equalization, limits local contrast amplification (e.g. threshold set to 40) to avoid excessive noise enhancement, processes images in blocks (e.g. 8x8 pixel blocks), enhances low-light area details, suitable for face image illumination normalization.

[0067] MobileNetV3: Lightweight Convolutional Neural Network, combines depthwise separable convolution and SE attention mechanism, optimizes structure through NAS search, significantly reduces computational load while ensuring accuracy, commonly used for real-time face feature extraction tasks.

[0068] MediaPipe Face Mesh Model: MediaPipe Face Detection Model, Google's open-source real-time face mesh rendering solution, based on binocular vision or monocular camera, can detect 468 facial key points, achieve high-precision face posture and expression analysis.

[0069] AdaptiveBaseline: Dynamic Baseline Correction, learns individual physiological feature distribution through Gaussian Mixture Model (GMM), dynamically updates baseline with confidence mechanism, adapts to different drivers' fatigue detection benchmarks.

[0070] partial_fit: Incremental Learning Method, used to update Gaussian Mixture Model parameters online, directly fits new data at low confidence, updates mean value through sliding window smoothing at high confidence, improves model stability.

[0071] EAR: Eye Aspect Ratio, measures the degree of eye opening, the smaller the value, the longer the eyes are closed, is one of the core visual features of fatigue detection.

[0072] MAR: Mouth Opening Ratio, monitors the degree of mouth opening, the larger the value, the more likely yawning or jaw relaxation, reflects the state of nervous fatigue.

[0073] reset: Reset Method, used to clear baseline model parameters and confidence, suitable for user switching or environmental mutation scenarios, reinitialize fatigue detection benchmarks.

[0074] In the related art, the technical report of the International Society of Automotive Engineers indicates that the traditional visual solution has a face feature extraction completeness rate of less than 45% in strong light or night scenes with a contrast of less than 0.3, resulting in a false detection rate of up to 28% (SAE Technical Paper 2021-01-0732). The commonly used fixed threshold strategy cannot adapt to the physiological differences of drivers. An article published in the top journal of computer graphics in 2019 indicates that the baseline EAR value of people with drooping eyelids is 23% lower than that of normal people, the MAR detection of myopic people is invalid due to the reduced dynamic range of the pupil, and the head posture baseline of special professional drivers such as long-distance truck drivers is offset by 18°±5 (IEEE TVCG 2019). When running the traditional solution on a vehicle-grade embedded platform such as the Qualcomm Snapdragon 8155 chip, the edge computing resources are limited, the peak memory occupancy is 512 MB, which is 3 times the regular configuration of a vehicle terminal, the single-frame processing delay is ≥80 ms, which is difficult to meet the 100 ms level real-time requirement, and the floating-point operation amount exceeds the upper limit of the computing power of the ARM Cortex-A76 core. The 28th World Congress on Intelligent Transportation Systems held in 2022 indicates that in complex scenes such as mask obstruction with coverage ≥30% or head rotation with angular velocity ≥30° / s, the comprehensive detection accuracy of existing solutions decreases to 58% (ITS World Congress 2022 data).

[0075] Therefore, in the embodiments of the present application, an edge fatigue detection method based on adaptive threshold adjustment is provided, which relates to the technical field of intelligent detection. The edge fatigue detection method based on adaptive threshold adjustment provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, and the like, but is not limited thereto; the server end can be configured as a standalone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements the edge fatigue detection method based on adaptive threshold adjustment, and the like, but is not limited to the above forms.

[0076] The application is operable in a variety of general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0077] Figure 1 is an optional flowchart of the edge fatigue detection method based on adaptive threshold adjustment provided by an embodiment of the application, Figure 1 The method in the method can include but is not limited to including steps S110 to S150:

[0078] Step S110: Obtain facial image data and facial infrared data of a real-time state of a driver;

[0079] Step S120: Preprocess the facial image data and the facial infrared data;

[0080] Step S130: Locate facial landmark points of the preprocessed facial image data and the facial infrared data through a facial feature detection model, to obtain an enhanced 3D landmark point image;

[0081] Step S140: Compare the enhanced 3D landmark point image with a physiological feature database, to obtain physiological data;

[0082] Step S150: Calculate the physiological data through a dynamic threshold decision model and a multi-index fusion engine, to obtain a fatigue decision.

[0083] In some embodiments, the face image data and the face infrared data of the driver in the cockpit are acquired in real time by the sensor, and the acquired data is preprocessed, the preprocessed acquired data is processed by a face feature detection model, the acquired data is clear and can reflect the landmark points of the face features of the driver for fatigue detection, that is, an enhanced 3D landmark point image is obtained. The face feature detection model solves the detection failure problem of the traditional scheme in the extreme light, head posture change and occlusion scene, enhances the local contrast in dark light environment, and makes the accuracy rate of the weak light scene increase from 37.5% to 82.6%. It supports accurate tracking within a head posture angle of ± 50°, and combines infrared thermal imaging and visible light images. The detection rate under the mask occlusion scene is still 76.4%. The enhanced 3D landmark point image is compared with the physiological feature database trained in advance, and the physiological data for calculating and judging fatigue decision are obtained. The physiological feature database integrates three-dimensional indicators of ECR, MOR and HNFR, which better cope with false positives and false negatives in complex driving scenes. The physiological data is calculated by a dynamic threshold decision model and a multi-index fusion engine, so as to obtain a fatigue decision. The fatigue decision directly reflects whether the current state of the driver is fatigue driving, so as to ensure the safety of the driver on the road. The dynamic threshold decision model eliminates the false detection problem caused by the fixed threshold, and adapts to the physiological feature differences of the driver. In some embodiments, in step S110, a camera and an infrared thermal imaging module are arranged at the driving position. The camera is a visible light camera with a resolution of ≥ 1080P@30fps, to obtain camera RGB data, i.e. face image data. The frame rate of the infrared thermal imaging module is ≥ 25fps, to obtain infrared thermal imaging data, i.e. face infrared data. The position and angle of installation should ensure that the face image of the driver can be obtained at all times.

[0084] In some embodiments, in step S120, the face image data and the face infrared data are preprocessed, and the preprocessing includes time synchronization and spatial alignment.

[0085] Specifically, the face image data and the face infrared data are marked with accurate time stamps and aligned by a dynamic time warping algorithm. The nose tip, eye corner and other reference points are located by face key point detection, and the two image coordinates of the face image data and the face infrared data are unified by affine transformation to ensure consistent feature space positions.

[0086] In some embodiments, in step S130, the face feature detection model includes a contrast-limited adaptive histogram equalization model and a MediaPipe face detection model. The contrast-limited adaptive histogram equalization model, i.e., CLAHE preprocessing enhancement, is used to perform histogram equalization on the input image blocks to improve the contrast in dark light environments. Experiments show that CLAHE preprocessing enhancement improves the feature point detection accuracy in weak light scenes by 45%. The MediaPipe face detection model, i.e., MediaPipeFace Mesh model, extracts features through a lightweight convolutional neural network backbone network and combines a graph convolution network to predict 3D face landmark points. This enables sub-millimeter level face landmark point positioning in complex lighting and pose conditions. The core formula is:

[0087] Q = GraphConvolution(f MobileNetV3 (I));

[0088] where I represents the input image, f() represents a feature extraction function composed of a MobileNetV3 network, GraphConvolution() represents a graph convolution operation, and Q e R468x3 represents three-dimensional key point coordinates.

[0089] In some embodiments, as shown in Figure 2 , Figure 2 is a flowchart of data processing of the contrast-limited adaptive histogram equalization model, Figure 2 The method in the above embodiment can include but is not limited to the following steps S131-S136:

[0090] Step S131: dividing the preprocessed face image data and face infrared data into a plurality of non-overlapping small blocks;

[0091] Step S132: calculating the pixel gray value in each small block to obtain the histogram of each small block;

[0092] Step S133: clipping the histogram according to the pixel threshold to obtain a contrast-limited histogram;

[0093] Step S134: performing normalization processing on the clipped contrast-limited histogram to calculate the cumulative distribution function of each gray level and obtain a new gray mapping relationship;

[0094] Step S135: performing smoothing processing on adjacent small blocks by overlapping the blocks or fusing the information of the contrast-limited histograms of adjacent regions in the interpolation process;

[0095] Step S136: mapping the pixel gray value in each small block back to the original image coordinate system through the new gray mapping relationship to obtain an enhanced image.

[0096] Specifically, as Figure 3 shown, Figure 3 is an algorithm flowchart of a contrast-limited adaptive histogram equalization model enhancement process, and the following process is obtained according to the algorithm flowchart. The input image is divided into a plurality of non-overlapping small blocks, usually rectangular regions of 8x8 or 16x16 pixels, and the blocking helps to make local adjustments to the brightness distribution of different regions of the image, avoiding the destruction of local details by global histogram equalization. The histogram of the pixel gray value in each block is calculated, and the number of pixels of each gray level is counted. The histogram range is usually limited to 0-255 (8-bit image), but can be adjusted according to actual needs, to obtain the histogram of each small block. The histogram is cut to limit the maximum enhancement amplitude of the contrast in each block. If the number of pixels of a certain gray level exceeds the preset threshold, the excess pixels are evenly distributed to other gray levels to prevent overexposure or noise amplification, and a contrast-limited histogram is obtained. The contrast-limited histogram is normalized to redistribute the pixel values to expand the local contrast, and the original gray value is mapped to a new gray range through the cumulative distribution function (CDF) to enhance the details of the dark or bright area, and a new gray mapping relationship is obtained. To reduce the blocking effect caused by blocking, the histograms of adjacent blocks are smoothed, and common methods include overlapping blocks or fusing the histogram information of adjacent regions in the interpolation process. The pixel values of all blocks after equalization are remapped back to the original image coordinate system, and all blocks are combined to generate the final enhanced image, which retains the local contrast while maintaining overall visual consistency, and an enhanced image is obtained.

[0097] In some embodiments, as Figure 4 shown, Figure 4 is a flowchart of data processing of a MediaPipe face detection model, Figure 4 The method in the method can include but is not limited to including steps S137 to S139:

[0098] Step S137: performing multi-scale feature extraction on the enhanced image through a lightweight convolutional neural network to obtain a real-time feature map;

[0099] Step S138: performing acceleration processing on the real-time feature map according to a heterogeneous acceleration hardware;

[0100] Step S139: performing prediction on the real-time feature map through a graph convolutional network to obtain an enhanced 3D landmark point image.

[0101] In some embodiments, in step S137, a 3-channel color image with an input resolution of 192*192 pixels, i.e., an enhanced image, is used as the original input data for the model, providing visual information for subsequent feature extraction. By reducing computational load or adapting to hardware acceleration, a fixed resolution can unify the input size, facilitating efficient processing by the neural network. The channels correspond to red, green, and blue pixel values, used to capture the color and texture features of the image. A lightweight convolutional neural network, MobileNetV3, is used to extract features from the input image. Through convolutional layers, activation functions, and pooling layers, basic features such as edges and corners, as well as high-level semantic features such as facial contours and organ structures, are extracted. Through optimization techniques such as depthwise separable convolution and bottleneck structures, the computational load is reduced while maintaining feature expressive power, adapting to the real-time inference needs of mobile or embedded devices. A feature map containing spatial location and semantic information, i.e., a real-time feature map, is generated for further processing by subsequent network layers.

[0102] In some embodiments, step S138 accelerates model inference using heterogeneous computing hardware, such as graphics processing units (GPUs), digital signal processors (DSPs), and neural network processing units (NPUs). Computational tasks are allocated based on device hardware characteristics; for example, GPUs excel at parallel computing, while NPUs are optimized for neural networks. This significantly improves model running speed, achieving millisecond-level real-time processing. It avoids the performance bottleneck of single-CPU processing, reduces power consumption, and extends device battery life, which is especially crucial for mobile devices such as smartphones and AR glasses. For cross-platform compatibility, it supports different hardware architectures, enabling the model to be deployed on various devices, such as smartphones, computers, and edge computing devices.

[0103] In some embodiments, step S139 outputs 468 3D key points. Specifically, feature information is processed through a graph convolutional network to predict the 3D coordinates (X, Y, Z) of 468 facial key points. Facial key points are treated as "nodes" in a graph, and the graph convolutional network captures the spatial relationships between nodes, such as geometric constraints like eye spacing and nose bridge height, to infer the 3D structure of the face. Based on 2D image features extracted from MobileNetV3, i.e., the implementation feature map, combined with the spatial reasoning capability of the graph convolutional network, 3D key point coordinates with depth information are generated, achieving stereo modeling of the facial mesh. Outputting 468 3D key points can accurately describe facial contours, corners of the eyes, corners of the mouth, and other details, thus enhancing the 3D landmark image. The MediaPipe face detection model combines the feature extraction capability of convolutional neural networks and the structural reasoning capability of graph convolutional networks, achieving efficient and real-time facial 3D key point detection through hardware acceleration.

[0104] In some embodiments, such as Figure 5 As shown, Figure 5 This is a flowchart of the process of building a physiological characteristic database.

[0105] Figure 5 The method may include, but is not limited to, steps S210 to S250:

[0106] Step S210: Estimate the probability density of the historical enhanced 3D landmark image data obtained after processing the historical data using a Gaussian mixture model to obtain the basic Gaussian mixture model parameters;

[0107] Step S220: Incrementally train the Gaussian mixture model by continuously inputting new historical enhanced 3D landmark image data, and update the parameters of the Gaussian mixture model until the confidence level reaches the threshold.

[0108] Step S230: When the confidence level reaches the threshold, calculate the normal physiological range boundary based on the Gaussian mixture model to obtain the physiological feature threshold table;

[0109] Step S240: Through correlation analysis, the physiological feature threshold table and the associated physiological information are fused to obtain a multidimensional physiological feature vector;

[0110] Step S250: Store the multidimensional physiological feature vectors according to the spatiotemporal index structure to obtain the physiological feature database.

[0111] In some embodiments, in steps S210 to S250, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the source code for the physiological characteristic threshold table. Specifically, facial image data and facial infrared data obtained from historical data are preprocessed, and then contrast is enhanced and illumination and temperature noise is eliminated using the CLAHE algorithm. MobileNetV3 is used to extract facial texture features, and MediaPipe Face Mesh is used to locate 468 key points (such as the corners of the eyes, the tip of the nose, and the chin) to obtain historical enhanced 3D landmark image data. Core indicators such as EAR (Eye-tracking aspect ratio), MAR (Mouth-opening ratio), and HNFR (Head-Nose Frequency) are calculated from the key point coordinates.

[0112] Feature values ​​such as EAR and MAR, along with driver information, are extracted, and physiological signals such as heart rate and steering wheel grip force are collected simultaneously. These feature values ​​are standardized and input into a Gaussian mixture model, initialized with three Gaussian components corresponding to three states: alertness, mild fatigue, and severe fatigue. The mean vector, covariance matrix, and weight coefficients of each component are estimated using an expectation-maximization algorithm to form the basic model parameters.

[0113] New historical enhanced 3D landmark point image data is continuously collected, and after feature extraction, it is input into the Gaussian mixture model for incremental training. Each time the partial_fit method is called, the system checks the current confidence: if it is lower than 0.9, the Gaussian mixture model parameters are directly updated and the confidence is improved; if it reaches 0.9, it is switched to a sliding window mean filtering mechanism to retain the historical mean value with a weight of 0.9 and integrate the new data mean value with a weight of 0.1, avoiding excessive fitting of the model due to short-term abnormal fluctuations. When the model parameters converge and the confidence is stable, the normal range boundary of the physiological characteristics is calculated based on the mean value ± 2.5 times the standard deviation of each Gaussian component, and the physiological characteristic threshold table is obtained.

[0114] The historical physiological characteristic threshold table is sorted by timestamp to form a queue. Exponential decay weights are used to aggregate the threshold values in the queue: the latest threshold value is given the highest weight, and the older threshold values decrease exponentially. This mechanism enables the threshold table to dynamically track the slow changes in the driver's physiological characteristics, such as fatigue tolerance improvement, while maintaining sensitivity to sudden abnormalities. Correlation analysis is performed on EAR, MAR, heart rate variability, blink frequency, etc. to calculate the correlation coefficient between the characteristics. Strongly correlated features are selected to construct a multi-dimensional feature vector space. For example, EAR, MAR, and HRV are combined into a three-dimensional vector with weights of 4:3:3, and the threshold values for each dimension come from the physiological characteristic threshold table generated in step S230. Z-score standardization is performed on each dimension to eliminate dimension differences, forming a unified feature expression space that allows data from different modalities to be compared and matched in the same space. Spatial indexing is used for the physiological characteristic database, and KD-tree indexing is established for the spatial coordinates of the 3D landmarks, supporting fast spatial neighborhood queries for locating the most similar historical data to the current pose. Through a three-level index structure, the physiological characteristic vectors corresponding to the enhanced 3D landmark point images are organized into an efficient and queryable database.

[0115] The 3D landmark point image obtained by real-time collection and processing is extracted to obtain a feature vector, and the Mahalanobis distance between the feature vector and the nearest N vectors in the database is calculated. If the distance is less than 2.0, it is determined to be "awake"; between 2.0 and 3.0, it is "fatigue"; and more than 3.0, it triggers an "abnormal" warning. At the same time, the system returns the feature dimension with the largest abnormal contribution (such as "EAR deviates from the baseline by 2.8σ") and labels the corresponding 3D landmark position (such as the eye corner key point). The comparison results, together with metadata such as timestamp and confidence, are stored in a result table to form a complete physiological state change record, supporting subsequent analysis and model optimization.

[0116] In some embodiments, a redundancy check process is also included in the abnormal state self-check mechanism, which is triggered when the detection results of five consecutive frames are contradictory. Specifically, the statistical distance between the current physiological characteristics and the baseline characteristics is calculated by monitoring Mahalanobis distance. Gradient limitation is implemented by setting a threshold of ≤5% for maximum gradient change to prevent parameter abrupt changes. When the Mahalanobis distance is greater than 2.5, the threshold update is frozen, and a manual review process is initiated. This mechanism ensures system robustness through the dual protection of Mahalanobis distance and gradient limitation.

[0117] In some embodiments, when camera and infrared thermal imaging module malfunctions causing sensor abnormalities, feature extraction failures, or data contamination leading to data abnormalities, sudden physiological abnormalities, dangerous driving behaviors, or system malfunctions, resulting in abnormal judgments under no-fatigue conditions, data validity verification is required to determine the specific situation. Specifically, data validity verification involves: acquiring enhanced 3D marker image data of the driver through the camera and infrared thermal imaging module, and simultaneously collecting timing signals from devices such as the steering wheel torque sensor or inertial measurement unit. Visual features such as eye-tracking aspect ratio (EAR), mouth opening ratio (MAR), and head pitch or yaw angles are extracted from the images; facial skin temperature distribution (e.g., temperature change trends around the eyes and nose) is extracted from the infrared data; and the time series of steering wheel torque fluctuations and head posture changes are analyzed from the sensor data.

[0118] like Figure 7 As shown, Figure 7 This is a schematic diagram of the source code for the fatigue probability calculation. It calls the `initial_validation` function to calculate the fatigue probability score P based on the data collected in the first 30 minutes. f Calculate the normalized values ​​of EAR and MAR (range 0-1, higher values ​​indicate higher fatigue levels), and the abnormal head posture index (e.g., the percentage of durations with pitch angles exceeding 15°). Calculate the overall score according to the weighting formula. The weights of each index can be determined in advance. If P... f >0.3, is considered invalid data (indicating potential fatigue or abnormal interference during data collection); if P f If the value is ≤0.3, then proceed to multi-dimensional cross-validation.

[0119] When P f≤0.3 When entering multi-dimensional cross-validation, analyze the correlation between EAR or MAR abnormal period (such as EAR <0.2) and facial temperature change; if EAR anomaly is accompanied by sudden drop of eye temperature (possibly due to reduced heat dissipation caused by closing eyes), it is determined as an effective fatigue signal, otherwise it is considered as interference (such as sudden change of light causing EAR miscalculation). Check whether the head low (pitch angle >20°) is synchronized in time with the sudden drop of steering torque (such as holding force <5N). If synchronized, it may reflect the decline of driving ability caused by fatigue; if the head posture is abnormal but the torque is normal (such as the driver adjusting the sitting posture), it is determined as non-fatigue abnormality. If multi-dimensional cross-validation does not find fatigue evidence, such as visual abnormalities caused by environmental interference, the current physiological data (EAR, MAR, temperature, torque, etc.) is added to the physiological feature database, and the partial_fit of the baseline model is updated. If the verification fails, the collection time is increased by an exponential backoff strategy, and the fatigue probability calculation and cross-validation are re-executed. When the cumulative effective data ratio (effective time length / total collection time length) ≥70%, it is considered that the data quality meets the standard, and the baseline modeling process (such as Gaussian mixture model initialization) is started; if the effective ratio is still <70% after 3 consecutive extensions, an artificial review request (with original data segments, feature curves and verification results) is sent to the customer end, and the data effectiveness is marked by artificial marking, such as "equipment failure" or "driver abnormal behavior". It can be understood that if the driver is already fatigued in the initialization stage (such as driving at dawn or working all night), the baseline data will be contaminated, and the subsequent dynamic threshold calculation will be completely invalid.

[0120] When the verification process is passed and the cumulative effective data meets the modeling requirements (such as ≥1 hour of continuous non-fatigue data), the Gaussian mixture model is initialized based on the cleaned feature data, 3 Gaussian components are set to fit different sub-scenarios in the wakeful state (such as the baseline difference between daytime / night driving), and the expectation maximization algorithm is used to iteratively calculate the mean, covariance and other parameters, and finally an individual initial baseline threshold table is generated (such as EAR normal range [0.3, 0.5]).

[0121] In some embodiments, in step S140, the 3D landmark image is enhanced to calculate eye movement aspect ratio (EAR), mouth opening ratio (MAR), head posture angle and other geometric feature values, and the temperature distribution data of infrared thermal imaging is extracted synchronously to form a multi-dimensional feature vector. The feature vector is input into the physiological feature database, and the historical data is quickly retrieved based on the time and space index structure of the database. The deviation degree data of each physiological indicator is obtained by calculating the Mahalanobis distance between the real-time feature vector and the baseline model parameters (such as the mean and covariance of the Gaussian mixture model) in the database, such as the difference between EAR and the baseline mean, the fluctuation range of the head posture angle, etc. This process only completes data comparison and generates physiological data containing feature values, deviation degrees and time stamps.

[0122] In some embodiments, in step S150, the real-time physiological data obtained from the comparison with the physiological feature database is compared with the double-layer dynamic threshold stored in the physiological feature database. The specific formula of the double-layer dynamic threshold is:

[0123] T adaptive = a · T global + (1 - a) · (μ personal + 2.5σ personal );

[0124] wherein a represents a global threshold weight, μ personal represents a personal baseline threshold, σ personal is a standard deviation, T global represents a global threshold, and T adaptive represents a dynamic threshold.

[0125] The global threshold is suitable for the basic standard of all drivers, and the personal baseline threshold is calculated based on the Gaussian mixture model parameters trained from the historical data of the driver. For each physiological indicator, if the real-time data exceeds both the global threshold and the individual threshold, the indicator is marked as “abnormal”; only the global threshold is exceeded, it is marked as “suspicious”; and neither is exceeded, it is marked as “normal”. The final integration of the marking results of all indicators forms the first fatigue judgment, which preliminarily determines whether the driver is currently in a fatigue state.

[0126] According to the result of the first fatigue judgment, Kalman filtering is used for smoothing processing to eliminate short-term fluctuations and misjudgments of the data. The Kalman filtering regards the fatigue judgment result as a state variable of a dynamic system, establishes a state transition model and an observation model, predicts the fatigue state at the next moment, and combines the real-time observation data (i.e. the first fatigue judgment result) for correction. For example, if the EAR indicator temporarily triggers an anomaly at a certain moment, but the Kalman filtering predicts that it belongs to the normal fluctuation range, the anomaly mark is corrected to “suspicious” or “normal”. Through continuous iteration of prediction and correction, a more stable and reliable second fatigue judgment is output, reducing false positives caused by noise or accidental factors.

[0127] The second fatigue judgment result is normalized and standardized for subsequent multi-index fusion. The preprocessed second fatigue judgment result is input into the multi-index fusion engine, which is integrated according to three dimensions of eye movement (ECR), mouth (MOR), and head posture (HNFR). The formula of the multi-index fusion engine is:

[0128]

[0129] The multi-index fusion engine determines whether to trigger a fatigue warning according to a preset threshold (such as ECR ≥ 50%) for each dimension (such as ECR). The logical "or" strategy is used to integrate the judgment results of the three dimensions, that is, as long as any one of the dimensions of ECR, MOR, and HNFR reaches the warning threshold, it is determined as a suspected fatigue state. In the experimental group scene, the weight coefficients of ECR, MOR, and HNFR are dynamically adjusted based on the baseline data of the driver (such as the correlation between historical fatigue states and each index). For example, if a driver is more likely to have a low head (HNFR abnormality) when he is tired, the weight proportion of HNFR is increased (such as from 30% to 40%). The logical "or" judgment result is combined with the weight to calculate the final fatigue score. If the comprehensive score is greater than or equal to the threshold, it is determined as a fatigue state; otherwise, it is not a fatigue state, that is, a fatigue decision. The multi-index fusion engine uses a space-time optimization algorithm, through parallel computing and pipeline processing, such as decomposing the detection process into three-level cache queues of preprocessing, feature extraction, and decision, and a lightweight model, to achieve a real-time inference speed of 54.3FPS, which is 37% higher in accuracy than the traditional single-index method.

[0130] In some embodiments, an edge computing optimization model applied to the overall architecture is also included to break through the computing power limit of the vehicle terminal and meet the real-time demand. Among them, MobileNetV3 lightweight architecture is adopted, which compresses the model volume to 3.6MB, which is 5.8% of the original model of 60MB; 4-bit hybrid quantization technology is applied to reduce 70% of the calculation amount while maintaining 95% accuracy; through the spatial dynamic ROI focusing algorithm, the face detection area is reduced from the full image to 42%; through the parallel processing pipeline technology, the detection process is divided into three-level cache queues of image preprocessing, feature extraction, and decision inference, and the processing speed is improved to 54.3FPS.

[0131] In some embodiments, the method can be deployed across platforms, implemented through the MediaPipe framework, model quantization technology, and standardized interface design. Specifically, the MediaPipe framework natively supports heterogeneous computing of GPUs (OpenGL / Vulkan), DSPs (Hexagon), and NPUs (Ascend). The model quantization technology adopts 4-bit hybrid quantization, reducing the model volume by 54%. The standardized interface design provides a unified API interface, adapting to Android, iOS, and embedded systems. The measured data shows that an end-to-end delay of 55 ms is achieved on a Snapdragon 8155 chip, verifying the cross-platform OTA update relying on the following technologies: MobileNet architecture + hybrid quantization to make the model lightweight from 62 MB to 3.6 MB, decoupling of detection logic and model, modular design supporting dynamic loading of new model files, and transmission of only model parameter increments to reduce OTA data volume. The experiment shows that the GPU memory occupancy is reduced to 186 MB after optimization, meeting the OTA update conditions of differential upgrading for vehicle terminals.

[0132] As shown in Figure 8 An edge fatigue detection device based on adaptive threshold adjustment, the device comprising:

[0133] A data acquisition module, the data acquisition module being configured to acquire facial image data and facial infrared data of a real-time state of a driver;

[0134] A data processing module, the data processing module being configured to pre-process the facial image data and the facial infrared data, locate facial landmark points of the pre-processed facial image data and the facial infrared data through a facial feature detection model, and obtain an enhanced 3D landmark point image;

[0135] A data decision module, the data decision module being configured to compare the enhanced 3D landmark point image with a physiological feature database to obtain physiological data, and calculate the physiological data through a dynamic threshold decision model and a multi-index fusion engine to obtain a fatigue decision.

[0136] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0137] Embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above edge fatigue detection method based on adaptive threshold adjustment when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0138] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0139] The present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above adaptive threshold adjustment based edge fatigue detection method.

[0140] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments, the present storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0141] The embodiments described in the present application are for more clearly illustrating the technical solutions of the present application, and do not constitute a limitation on the technical solutions provided by the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present application are also applicable to similar technical problems.

[0142] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0143] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above figures are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0144] It should be understood that, in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0145] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0146] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for edge fatigue detection based on adaptive threshold adjustment, characterized in that, The method comprises: acquiring facial image data and facial infrared data of a real-time state of a driver; preprocessing the facial image data and the facial infrared data; locating facial landmark points of the preprocessed facial image data and the facial infrared data through a facial feature detection model to obtain an enhanced 3D landmark point image; comparing the enhanced 3D landmark point image with a physiological feature database to obtain physiological data; comparing the physiological data with a double-layer dynamic threshold to obtain a first fatigue judgment, smoothing the first fatigue judgment through Kalman filtering to obtain a second fatigue judgment, preprocessing the second fatigue judgment, performing logical combination and dynamic weight adjustment on the preprocessed second fatigue judgment to obtain a fatigue decision, and a calculation formula of the double-layer dynamic threshold being: ； wherein, represents a global threshold weight, represents a personal baseline threshold, represents a standard deviation, represents a global threshold, represents a dynamic threshold.

2. The method of claim 1, wherein, The facial feature detection model comprises a contrast-limited adaptive histogram equalization model and a MediaPipe face detection model.

3. The method of claim 2, wherein, The data processing process of the contrast-limited adaptive histogram equalization model comprises the following steps: dividing the preprocessed facial image data and the facial infrared data into a plurality of non-overlapping small blocks; calculating pixel gray values in each small block to obtain a histogram of each small block; cutting the histogram according to a pixel threshold to obtain a contrast-limited histogram; performing normalization processing on the cut contrast-limited histogram to calculate a cumulative distribution function of each gray level and obtain a new gray mapping relationship; performing smoothing processing on adjacent small blocks by overlapping block or fusing information of the contrast-limited histogram of adjacent regions in an interpolation process; mapping pixel gray values in each small block back to an original image coordinate system through the new gray mapping relationship to obtain an enhanced image.

4. The method of claim 3, wherein, The data processing process of the MediaPipe face detection model comprises the following steps: extracting multi-scale features of the enhanced image through a lightweight convolutional neural network to obtain a real-time feature map; performing acceleration processing on the real-time feature map according to heterogeneous acceleration hardware; predicting the real-time feature map through a graph convolutional network to obtain an enhanced 3D landmark point image.

5. The method of claim 1, wherein, The construction process of the physiological feature database comprises the following steps: performing probability density estimation on historical enhanced 3D landmark point image data obtained after preprocessing of historical data through a Gaussian mixture model to obtain basic Gaussian mixture model parameters; incrementally training the Gaussian mixture model by continuously inputting new historical enhanced 3D landmark point image data to update parameters of the Gaussian mixture model until a confidence reaches a threshold value; when the confidence reaches the threshold value, calculating a normal physiological range boundary based on the Gaussian mixture model to obtain a physiological feature threshold table; performing feature fusion on the physiological feature threshold table and auxiliary physiological information through correlation analysis to obtain a multi-dimensional physiological feature vector; storing the multi-dimensional physiological feature vector according to a space-time index structure to obtain the physiological feature database.

6. The method of claim 5, wherein, Further comprising data validity verification, including the following steps: acquiring the historical enhanced 3D landmark image data; non-fatigue verification of the physiological data through fatigue probability and multi-dimensional cross verification; updating the physiological data to the physiological feature database when the verification is passed; triggering abnormal processing, extending the collection time of acquiring the physiological data and repeating the non-fatigue verification when the verification is not passed, and sending a review request to the customer service end if it still fails.

7. The method of claim 1, wherein, Further comprising an edge computing optimization model, the edge computing optimization model comprising a lightweight neural network architecture, 4-bit hybrid quantization, spatial dynamic ROI focusing, and a parallel processing pipeline; the lightweight neural network architecture is used for volume compression of each model; the 4-bit hybrid quantization is used to reduce memory occupation and calculation amount; the spatial dynamic ROI focusing is used to dynamically adjust the detection area according to the face position; the parallel processing pipeline is used to realize multi-level task overlap execution.

8. An edge fatigue detection device based on adaptive threshold adjustment, characterized by, The device comprises: a data acquisition module for acquiring real-time face image data and face infrared data of a driver; a data processing module for pre-processing the face image data and the face infrared data, locating facial landmarks through a facial feature detection model, and obtaining enhanced 3D landmark images; a data decision module for comparing the enhanced 3D landmark images with a physiological feature database to obtain physiological data, comparing the physiological data with a double-layer dynamic threshold to obtain a first fatigue judgment, smoothing the first fatigue judgment through Kalman filtering dynamic adjustment to obtain a second fatigue judgment, preprocessing the second fatigue judgment, logically combining and dynamically adjusting the weights of the preprocessed second fatigue judgment to obtain a fatigue decision; and the calculation formula of the double-layer dynamic threshold is: ； wherein, represents a global threshold weight, represents a personal baseline threshold, represents a standard deviation, represents a global threshold, represents a dynamic threshold.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to realize the method of any one of claims 1 to 7.

Images