Electronic sentry system based on array radar, infrared and optics and use method

Abstract

The invention belongs to the intelligent security and detection technology, and particularly discloses an electronic sentry system based on array radar, infrared and optics and a using method, the electronic sentry system comprises a sensor part, a wireless networking module, an edge computing platform and a display module, the sensor part comprises a millimeter wave array radar, a thermal infrared camera and an optical camera, the millimeter wave array radar communicates with the edge computing platform through a network port. The thermal infrared camera and the optical camera communicate with the edge computing platform through a USB. The edge computing platform is electrically connected with the display module, the edge computing platform is a Jetson Orin NX development board and carries a Ubuntu system, and a face recognition algorithm, a radar sensing algorithm and an infrared and optical fusion algorithm are built in the Ubuntu system. According to the electronic sentinel system based on the array radar, infrared and optics and the use method, power consumption can be reduced, the temperature measurement area is corrected through the internal and external parameter matrix information of the camera, and the accuracy of body temperature detection of the camera is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent security and detection technology, and in particular to an electronic sentinel system based on array radar, infrared and optics and a method of using the system. Background Art

[0002] When the "electronic sentinel" was launched, it became increasingly popular due to its multi-functionality and high efficiency. It has obvious advantages in improving safety, reducing labor costs, and enhancing information management. However, in actual application, its lack of information fusion between multiple sensors has limited its performance and scenarios. For example, in the absence of human traffic, cameras and other equipment in the traditional "electronic sentinel" system are still in working condition, resulting in unnecessary increase in system power consumption, affecting the long-term stable operation of the equipment. In addition, due to the neglect of the calibration of the internal and external parameter matrices and non-co-optical axes between the optical camera and the thermal infrared camera, the information collected by the thermal infrared camera and the optical camera cannot be accurately fused, which leads to inaccuracy and misjudgment of the fused data, affecting the overall monitoring effect and the reliability of the judgment. Therefore, it is urgent to invent an intelligent "electronic sentinel" system that can automatically wake up all day and all day, with low power consumption and high precision. Summary of the invention

[0003] The purpose of the present invention is to provide an electronic sentinel system based on array radar, infrared and optics and a method of use. The system utilizes core devices such as thermal infrared cameras, optical cameras, millimeter-wave array radars, and artificial intelligence edge computing boards such as Jetson Orin NX, and is equipped with a fusion algorithm to realize an automatic wake-up, low-power, high-precision "electronic sentinel" system.

[0004] To achieve the above-mentioned purpose, the present invention provides an electronic sentinel system based on array radar, infrared and optics, including a sensor part, a wireless networking module, an edge computing platform and a display module. The sensor part includes a millimeter-wave array radar, a thermal infrared camera and an optical camera. The millimeter-wave array radar communicates with the edge computing platform through a network port, and the thermal infrared camera and the optical camera both communicate with the edge computing platform through a USB; the edge computing platform is electrically connected to the display module, and the edge computing platform is a Jetson Orin NX development board equipped with an Ubuntu system, and the Ubuntu system has built-in face recognition algorithm, radar perception algorithm and infrared and optical fusion algorithm.

[0005] Preferably, the millimeter wave array radar is used to sense target information of the target, including target distance, speed and direction, and to determine whether it is necessary to start a thermal infrared camera and an optical camera for target detection; the thermal infrared camera is used to measure the temperature of the target, the optical camera is used for face detection, the wireless networking module is used to transmit pictures to the cloud for detection when the network is good, the edge computing platform is used for local face recognition and infrared temperature measurement detection when the network is poor, and the display module is used to display the detection results and early warning prompt information.

[0006] The method for using the electronic sentry system based on array radar, infrared and optics includes the following steps: Step S1, sensing target information including direction, distance and speed through millimeter wave array radar; Step S2: Based on the information sensed by the millimeter wave array radar, use the radar sensing algorithm to determine whether the thermal infrared camera and the optical camera should be started. If not, proceed to step S1; otherwise, proceed to step S3; Step S3, starting the optical camera and the thermal infrared camera to collect images; Step S4: determine whether the network status is good. If the network is good, send the image to the cloud for detection. If the network is not good, detect it locally using the face recognition algorithm and the infrared and optical fusion algorithm.

[0007] Preferably, in step S1, the radar sensing algorithm uses a millimeter wave array radar to sense target information and obtain the speed, phase and distance of the target. The specific process is: Continuously modulate the millimeter wave array radar transmit signal, assuming the initial phase is , then the initial phase of the transmitted signal is expressed as: (1); in, t Indicates time, represents the integral parameter in the integral, T represents the period, Represents the instantaneous frequency of the transmitted signal. Since it is a linear frequency modulation signal, represents the initial frequency, μ is the linear frequency modulation index; Then its transmitted signal is expressed as: (2); in, is a window function, which only takes one signal cycle, that is, the signal is valid in a time window [-T / 2, T / 2]; After the signal is transmitted, it is received after a delay, and the signal received by the radar is: (3); In the formula, is the received signal amplitude, which is related to the target scattering cross-sectional area and the signal attenuation coefficient. is the phase shift after reflection, Indicates the delay of the echo signal; According to the law of electromagnetic wave transmission, assuming that the moving speed of the object is , the distance from the launch point is , then the echo delay is: (4); Where c is the speed of light; The power divider of the continuous modulation millimeter wave array radar divides the original signal into two parts. One part of the signal is transmitted for target detection, and the other part of the signal is used as the local oscillator signal to mix with the echo signal. The mixed signal is of the same type as the echo signal. , the reference signal is: (5); The high-order terms are filtered out by a low-pass filter to obtain the beat signal: (6); Where * indicates the conjugate of the signal; The center frequency of the beat signal is divided into the frequency related to the target distance and the Doppler frequency caused by the moving target; The calculation formula of the frequency related to the target distance is: (7); The Doppler frequency caused by a moving target is: (8); definition For time delay The frequency caused is related to the distance of the moving target. is the frequency caused by the target Doppler frequency shift, which is related to the speed of the moving target. The difference frequency of the beat signal is expressed as , we get the following formula: (9); (10); The expressions of target distance and speed are further obtained as follows: (11); in, Indicates the delay time of the echo signal; (12); The collected beat signal data Perform a two-dimensional fast Fourier transform to obtain fast time frequency and slow time frequency , and then obtain the distance and speed information of the target.

[0008] Preferably, in step S4, the face recognition algorithm specifically includes the following steps: Step Sa1, constructing a face dataset; Step Sa2, improving the YOLOv8 algorithm model; Step Sa3, training the improved YOLOv8 algorithm model; Step Sa4, establishing a database; Step Sa5, training FaceNet model; Step Sa6: Algorithm edge platform deployment and acceleration; Step Sa7, input the output features of the YOLOv8 algorithm model into FaceNet for feature extraction and mapping; Step Sa8: Compare the mapped facial feature vector with the facial image in the database to obtain the final result.

[0009] Preferably, step Sa1 specifically comprises: constructing a data set using public or collected face data as training samples; Step Sa2 is as follows: replace the backbone network of the YOLOv8 algorithm model with FasterNet, and trim the shallow small target detection head layer; Step Sa3 is specifically as follows: using the face data set constructed in step Sa1 to train the improved YOLOv8 algorithm model, learn the features of the face, and accurately recognize the face; The database established in step Sa4 is a database that needs to store the recognized faces; Step Sa5 is specifically as follows: using VGGFace2 or CASIA-WebFace large face datasets to train the FaceNet algorithm model; Step Sa6 is as follows: deploy the built and trained YOLOv8 algorithm model to the edge computing platform, and use TensorRT acceleration technology and INT8 shaping quantization technology to accelerate the running speed of the face recognition algorithm on the edge computing platform; Step Sa7 is specifically as follows: using the YOLOv8 algorithm model to detect the face, then cropping the local face part for processing, and inputting the processed face image into the FaceNet network for further feature extraction and spatial mapping; Step Sa8 is specifically as follows: performing Euclidean distance calculation between the feature vector mapped in step Sa6 and the face vector in the database to determine the final face category.

[0010] Preferably, in step S4, the specific process of the infrared and optical fusion algorithm is: According to the conversion relationship of the coordinate system, the conversion from the world coordinate system to the pixel coordinate system is as follows: (13); in, , , is the three-dimensional coordinates of the x-axis, y-axis, and z-axis of a point in the world coordinate system, is the depth of the target from the camera coordinate plane; make For the matrix , that is, the camera intrinsic parameter matrix; For the matrix , that is, the camera extrinsic matrix; Under the thermal infrared camera and optical camera, the conversion of the target world coordinates to pixel coordinates is shown in formulas (14) and (15) respectively: (14); (15); in, , They are the camera internal and external parameters of thermal infrared imaging, , They are the intrinsic and extrinsic parameters of the optical camera respectively; According to the basic rules of matrix operations and formulas (14) and (15), the pixel coordinates under the infrared camera and the pixel coordinates under the optical camera are converted to: (16); Among them, the intrinsic and extrinsic parameters of the thermal infrared camera and optical camera are measured by Zhang Zhengyou's marking method.

[0011] Therefore, the present invention adopts the above-mentioned electronic sentinel system based on array radar, infrared and optics and the method of use, and the beneficial effects are as follows: (1) Compared with the existing technology and the original algorithm, the improved YOLOv8 algorithm model proposed in the present invention has obvious improvements in model size and algorithm effect.

[0012] (2) The improved algorithm proposed in this invention outperforms other models in terms of accuracy, recall rate, average accuracy, floating-point operations and number of parameters on the provided data set.

[0013] (3) The present invention aims to solve the problem of excessive power consumption caused by the camera and image transmission still working in scenes without people. The millimeter-wave array radar is used to detect moving targets in the control area, and the direction of arrival (DOA) is estimated to measure the direction, the speed is measured by Doppler shift, and the distance is measured by time delay. The direction, speed and distance of the target are accurately obtained to determine whether it is necessary to wake up the optical camera, thermal infrared imaging, image transmission and other sensors to start working, thereby reducing power consumption.

[0014] (4) The present invention solves the problem of inaccurate image fusion and deviation between the temperature measurement area and the actual face area caused by not considering the internal and external parameter matrices of the optical camera and the thermal infrared camera. The present invention uses the information of the internal and external parameter matrices of the camera to correct the temperature measurement area, thereby improving the accuracy of the thermal infrared camera in detecting body temperature.

[0015] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall architecture of an embodiment of an electronic sentinel system based on array radar, infrared and optics of the present invention; Figure 2 It is a system overall flow chart of an embodiment of a method for using an electronic sentinel system based on array radar, infrared and optics of the present invention; Figure 3 It is a flow chart of a face recognition algorithm of an embodiment of a method for using an electronic sentinel system based on array radar, infrared and optics of the present invention; Figure 4 It is a structural diagram of an improved YOLOv8 algorithm model according to an embodiment of a method for using an electronic sentinel system based on array radar, infrared and optics of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0018] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0019] like Figure 1 As shown, the electronic sentry system based on array radar, infrared and optics includes a sensor part, a wireless networking module, an edge computing platform and a display module. The sensor part includes a millimeter-wave array radar, a thermal infrared camera and an optical camera. The millimeter-wave array radar communicates with the edge computing platform through a network port, and the thermal infrared camera and the optical camera both communicate with the edge computing platform through a USB. The edge computing platform is electrically connected to the display module.

[0020] The millimeter wave array radar of this embodiment refers to a continuous modulated wave radar, and the wavelength of the radar is in the millimeter wave range. It is mainly used to sense the target information of the target, including the target distance, speed and direction, and to determine whether it is necessary to start the thermal infrared camera and the optical camera for target detection. The thermal infrared camera is used to measure the temperature of the target. The optical camera is used for face detection. The wireless networking module is used to transmit the picture to the cloud for detection when the network is good.

[0021] The edge computing platform is used for local face recognition and infrared temperature measurement when the network is not good. The display module is used to display the detection results and early warning information.

[0022] The edge computing platform is the NVIDIA Jetson Orin NX development board, and is equipped with the Ubuntu 20.04 system. The Ubuntu 20.04 system has built-in face recognition algorithms, radar perception algorithms, and infrared and optical fusion algorithms, and finally displays the recognition results through the display.

[0023] The face recognition algorithm of the present invention refers to a face recognition method using the improved YOLOv8 and FaceNet; the radar perception algorithm refers to a method of sensing target information by measuring the direction through angle of arrival (DOA) estimation, measuring the speed through Doppler shift, and measuring the distance through time delay according to the working principle of array radar; the infrared optical fusion method refers to a method of fusing the results of two sensors at the decision layer by using the principle of coordinate system conversion.

[0024] The angle of arrival (DOA) estimation measurement direction refers to that for an array radar, under the condition of assuming a far-field model, the wavefront of the electromagnetic wave is a plane when it reaches each antenna. Depending on the array arrangement, the arrival time, the angle information of the electromagnetic wave reflection signal, that is, the angle information of the object, can be calculated based on the arrangement distance and time difference of the array antenna.

[0025] Doppler shift refers to the fact that for a moving object, the frequency of the electromagnetic wave will change accordingly depending on the speed, thereby measuring the speed.

[0026] Using time delay to measure distance means that for objects at different distances, the time difference from the emission to the reception of electromagnetic waves is different. The distance can be obtained by multiplying the time difference between emission and reception by the speed.

[0027] like Figure 2 As shown, the method for using the electronic sentry system based on array radar, infrared and optics includes the following steps: Step S1, sensing target information including direction, distance and speed through millimeter wave array radar; Step S2: Based on the information sensed by the millimeter wave array radar, use the radar sensing algorithm to determine whether the thermal infrared camera and the optical camera should be started. If not, proceed to step S1; otherwise, proceed to step S3; Step S3, starting the optical camera and the thermal infrared camera to collect images; Step S4, determine whether the network status is good. If the network is good, send the image to the cloud for detection. If the network is not good, detect it locally through the face recognition algorithm and the infrared and optical fusion algorithm. Since the cloud has large computing power and fast detection speed, in order to maximize the detection speed, it is sent to the cloud for detection when the network is good. However, in order to cope with emergencies such as poor network and ensure the working stability of the system, local detection is also set.

[0028] The principle of the radar perception algorithm of this embodiment is: using the millimeter wave array radar to perceive the target information and obtain the speed, phase and distance of the target. The specific process is as follows: Continuously modulate the millimeter wave array radar transmit signal, assuming the initial phase is , then the initial phase of the transmitted signal can be expressed as: (1); in, t Indicates time, represents the integral parameter in the integral, T Represents the cycle, Represents the instantaneous frequency of the transmitted signal. Since it is a linear frequency modulation signal, represents the initial frequency, μ is the linear frequency modulation index; Then its transmitted signal is expressed as: (2); in, is a window function, which only takes one signal cycle, that is, the signal is valid in a time window [-T / 2, T / 2]; After the signal is transmitted, it is received after a certain delay. The signal received by the radar is: (3); In the formula, is the received signal amplitude, which is related to the target scattering cross-sectional area and the signal attenuation coefficient. is the phase shift after reflection, Indicates the delay of the echo signal; According to the law of electromagnetic wave transmission, assuming that the moving speed of the object is , the distance from the launch point is , then the echo delay is: (4); in, c is the speed of light; The power divider of the continuous modulation millimeter wave array radar divides the original signal into two parts. One part of the signal is transmitted for target detection, and the other part of the signal is used as the local oscillator signal to mix with the echo signal. The mixed signal is of the same type as the echo signal. , the reference signal is: (5); And by filtering out the high-order terms through a low-pass filter, the beat signal can be obtained: (6); Where * indicates the conjugate of the signal; The center frequency of the beat signal is divided into two parts, one is the frequency related to the target distance, and the other is the Doppler frequency caused by the moving target; The calculation formula of the frequency related to the target distance is: (7); The Doppler frequency caused by a moving target is: (8); definition For time delay The frequency caused is related to the distance of the moving target. is the frequency caused by the target Doppler frequency shift, which is related to the speed of the moving target. The difference frequency of the beat signal can be expressed as , therefore, the following formula can be obtained: (9); (10); Further, we can derive the expressions of target distance and speed: (11); in, Indicates the delay time of the echo signal; (12); Finally, the collected intermediate frequency signal data (mixed signal of the transmitting signal and the receiving signal) is the beat signal data. Perform a two-dimensional fast Fourier transform to obtain fast time frequency and slow time frequency , and then the distance and speed information of the target can be obtained.

[0029] like Figure 3 As shown, the face recognition algorithm of this embodiment specifically includes the following steps: Step Sa1, constructing a face dataset; The public dataset or some collected face data can be used to construct a dataset as training samples for the subsequent YOLOv8 model.

[0030] Step Sa2, improving the YOLOv8 algorithm model; First, FasterNet is used to replace the backbone network of the YOLOv8 algorithm model to reduce the number of parameters and obtain the improved YOLOv8 algorithm model, as shown in Figure 4 As shown in the figure, the image first enters the FasterNet backbone network for feature extraction, and the extracted features are fused in the feature fusion network part to fuse the multi-level features to ensure that it can be sensitive to targets of different sizes. Finally, the middle-level target detection head and the large target detection head are separated to detect the features fused by the feature fusion network part; secondly, considering that in the actual face recognition process, the size of the face is relatively large, and there is no small target, because the detection will only be started when it is close to the electronic sentry system. Therefore, according to the characteristics of face detection, the shallow small target detection head layer is cropped to further reduce the model size, allowing the model to pay more attention to large targets, thereby greatly speeding up the running speed and further improving the face recognition effect, so that it can be deployed on edge computing platforms with limited computing power in the future.

[0031] Figure 3 The functions of each module are as follows: C2F is the CSP2 feature fusion module, which is used for feature extraction and fusion; Concat is a feature splicing module, which is used to splice multi-layer features in the channel dimension; Unsample is an upsampling module, which is used for feature map upsampling; CBS is a convolution-batch normalization-activation module, which is used to improve feature expression and normalize; Conv2d is a two-dimensional convolution module, which is used to extract features through convolution to generate classification and regression results; Bbox Loss is a bounding box loss, which is used to optimize the error between the predicted bounding box and the true bounding box; Cls Loss is a category loss, which is used to optimize the accuracy of the target prediction category.

[0032] Step Sa3, training the improved YOLOv8 algorithm; The face dataset constructed in step Sa1 is used to train the improved YOLOv8 algorithm model to learn the features of faces, thereby accurately identifying faces.

[0033] Step Sa4, establishing a database; The established database is a database that needs to store recognized faces. In this embodiment, a Mysql database is selected to store face images for subsequent result comparison.

[0034] Step Sa5, training FaceNet model; Use VGGFace2 or CASIA-WebFace large face datasets to train the FaceNet algorithm model; Step Sa6, algorithm edge platform deployment and acceleration, specifically: deploy the built and trained YOLOv8 algorithm model to the Jetson Orin NX edge computing platform, use TensorRT acceleration technology and INT8 shaping quantization technology to accelerate the running speed of the face recognition algorithm on the edge computing platform, and further accelerate the running speed of the algorithm at the edge.

[0035] Step Sa7, input the output features of the YOLOv8 algorithm model into FaceNet for feature extraction and mapping, specifically: use the YOLOv8 algorithm model to detect the face, then crop the local face part for processing, and input the cropped face part image into the FaceNet network for further feature extraction and spatial mapping.

[0036] Step Sa8, compare the mapped facial feature vector with the facial image in the database to obtain the final result, specifically: perform Euclidean distance calculation on the feature vector mapped in step Sa6 and the facial vector in the database to determine the final face category.

[0037] In the actual algorithm comparison ablation experiment, the improved algorithm proposed in this embodiment has obvious improvements in model size and algorithm effect compared with the existing patent and the original algorithm, as shown in Table 1.

[0038] Table 1 Comparative ablation experiment results

[0039] It can be seen from Table 1 that the improved YOLOv8 algorithm model of this embodiment has obvious advantages over other models in the face data set collected by itself. It can be seen that the YOLOv8+FasterNet improved algorithm proposed in this embodiment is better than other models in terms of accuracy, recall rate, average accuracy, floating-point operations and parameter quantity on the provided data set. That is, with the least number of parameters and the least number of floating-point operations, the accuracy, recall rate and average accuracy are not inferior to or even higher than other algorithm models.

[0040] For face detection, the present invention uses the improved YOLOv8 algorithm and FaceNet algorithm for face recognition. After the face is detected, the face recognition frame is obtained. According to the obtained target distance information, the thermal infrared image information can be fused into the optical recognition, and then the two can be fused to complete the integrated detection of face and body temperature. Specifically, the principle is as follows: After the optical face detection algorithm detects a face, it will output a detection frame of the face part. The detection frame has corresponding pixel coordinate values ​​in the pixel coordinate system. According to the coordinate system relationship between the thermal infrared camera and the pure optical camera, the optical face detection frame can be transformed to the position of the image pixel in the thermal infrared camera. The thermal infrared imaging performs thermal imaging detection on the face area corresponding to the thermal imaging camera, and the highest temperature is fed back to the optical imaging detection frame, thus completing the fusion of thermal infrared imaging and optical face detection. And due to the transformation of the coordinate system, the error caused by not considering the internal and external parameter matrix information of the camera is eliminated, making the infrared imaging temperature measurement result more accurate.

[0041] In this embodiment, the specific process of the infrared and optical fusion algorithm is as follows: According to the conversion relationship of the coordinate system, the conversion from the world coordinate system to the pixel coordinate system is as follows: (13); in, , , is the three-dimensional coordinates of the x-axis, y-axis, and z-axis of a point in the world coordinate system. is the depth of the target from the camera coordinate plane; make For the matrix , that is, the camera intrinsic parameter matrix; For the matrix , that is, the camera extrinsic matrix; Since the world coordinate system is unique, the position of the target in the world coordinate system remains unchanged under the thermal imaging camera and the optical camera. Under the imaging of the thermal infrared camera and the optical camera, the conversion of the target world coordinate to the pixel coordinate can be obtained as shown in formulas (14) and (15) respectively: (14); (15); in, , They are the camera internal and external parameters of thermal infrared imaging, , They are the intrinsic and extrinsic parameters of the optical camera respectively; According to the basic rules of matrix operations and formulas (14) and (15), the pixel coordinates under the infrared camera and the pixel coordinates under the optical camera can be converted into: (16); Among them, the internal and external parameters of the thermal infrared camera and optical camera are measured by Zhang Zhengyou's marking method, thus completing the conversion of the pixel coordinates of the optical camera imaging to the pixel coordinates under the thermal infrared camera.

[0042] Therefore, the present invention adopts the above-mentioned electronic sentry system based on array radar, infrared and optics and the use method, uses millimeter wave array radar to detect moving targets in the control area, measures the direction through angle of arrival DOA estimation, uses Doppler shift to measure the speed, and uses time delay to measure the distance, so as to accurately obtain the direction, speed and distance of the target, so as to determine whether it is necessary to wake up the optical camera, thermal infrared imaging, image transmission and other sensors to start working, thereby reducing power consumption.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An electronic sentinel system based on array radar, infrared and optics, characterized in that: It includes a sensor part, a wireless networking module, an edge computing platform and a display module. The sensor part includes a millimeter-wave array radar, a thermal infrared camera and an optical camera. The millimeter-wave array radar communicates with the edge computing platform through a network port, and the thermal infrared camera and the optical camera both communicate with the edge computing platform through a USB. The edge computing platform is electrically connected to the display module. The edge computing platform is a Jetson Orin NX development board and is equipped with an Ubuntu system. The Ubuntu system has built-in face recognition algorithms, radar perception algorithms, and infrared and optical fusion algorithms.

2. The electronic sentinel system based on array radar, infrared and optics according to claim 1 is characterized in that: The millimeter wave array radar is used to sense the target information of the target, including the target distance, speed and direction, and to determine whether it is necessary to start the thermal infrared camera and the optical camera for target detection; the thermal infrared camera is used to measure the temperature of the target, the optical camera is used for face detection, the wireless networking module is used to transmit the image to the cloud for detection when the network is good, the edge computing platform is used for local face recognition and infrared temperature measurement detection when the network is not good, and the display module is used to display the detection results and early warning prompt information.

3. The method for using the electronic sentinel system based on array radar, infrared and optics as described in any one of claims 1 to 2, characterized in that: The following steps are involved: Step S1, sensing target information including direction, distance and speed through millimeter wave array radar; Step S2: Based on the information sensed by the millimeter wave array radar, use the radar sensing algorithm to determine whether the thermal infrared camera and the optical camera should be started. If not, proceed to step S1; otherwise, proceed to step S3; Step S3, starting the optical camera and the thermal infrared camera to collect images; Step S4: determine whether the network status is good. If the network is good, send the image to the cloud for detection. If the network is not good, detect it locally using the face recognition algorithm and the infrared and optical fusion algorithm.

4. The method for using the electronic sentinel system based on array radar, infrared and optics according to claim 3 is characterized in that: In step S1, the radar sensing algorithm uses the millimeter wave array radar to sense the target information and obtain the speed, phase and distance of the target. The specific process is as follows: Continuously modulate the millimeter wave array radar transmit signal, assuming the initial phase is , then the initial phase of the transmitted signal is expressed as: （1）； in, t Indicates time, represents the integral parameter in the integral, T represents the period, Represents the instantaneous frequency of the transmitted signal. Since it is a linear frequency modulation signal, represents the initial frequency, μ is the linear frequency modulation index; Then its transmitted signal is expressed as: （2）； in, is a window function, which only takes one signal cycle, that is, the signal is valid in a time window [-T / 2, T / 2]; After the signal is transmitted, it is received after a delay, and the signal received by the radar is: （3）； In the formula, is the received signal amplitude, which is related to the target scattering cross-sectional area and the signal attenuation coefficient. is the phase shift after reflection, Indicates the delay of the echo signal; According to the law of electromagnetic wave transmission, assuming that the moving speed of the object is , the distance from the launch point is , then the echo delay is: （4）； Where c is the speed of light; The power divider of the continuous modulation millimeter wave array radar divides the original signal into two parts. One part of the signal is transmitted for target detection, and the other part of the signal is used as the local oscillator signal to mix with the echo signal. The mixed signal is of the same type as the echo signal. , the reference signal is: （5）； The high-order terms are filtered out by a low-pass filter to obtain the beat signal: （6）； Where * indicates the conjugate of the signal; The center frequency of the beat signal is divided into the frequency related to the target distance and the Doppler frequency caused by the moving target; The calculation formula of the frequency related to the target distance is: （7）； The Doppler frequency caused by a moving target is: （8）； definition For time delay The frequency caused is related to the distance of the moving target. is the frequency caused by the target Doppler frequency shift, which is related to the speed of the moving target. The difference frequency of the beat signal is expressed as , we get the following formula: （9）； （10）； The expressions of target distance and speed are further obtained as follows: （11）； in, Indicates the delay time of the echo signal; （12）； The collected beat signal data Perform a two-dimensional fast Fourier transform to obtain fast time frequency and slow time frequency , and then obtain the distance and speed information of the target.

5. The method for using the electronic sentinel system based on array radar, infrared and optics according to claim 4, characterized in that: In step S4, the face recognition algorithm specifically includes the following steps: Step Sa1, constructing a face dataset; Step Sa2, improving the YOLOv8 algorithm model; Step Sa3, training the improved YOLOv8 algorithm model; Step Sa4, establishing a database; Step Sa5, training FaceNet model; Step Sa6: Algorithm edge platform deployment and acceleration; Step Sa7, input the output features of the YOLOv8 algorithm model into FaceNet for feature extraction and mapping; Step Sa8: Compare the mapped facial feature vector with the facial image in the database to obtain the final result.

6. The method for using the electronic sentinel system based on array radar, infrared and optics according to claim 5, characterized in that: Step Sa1 specifically includes: constructing a data set using public or collected face data as training samples; Step Sa2 is as follows: replace the backbone network of the YOLOv8 algorithm model with FasterNet, and trim the shallow small target detection head layer; Step Sa3 is specifically as follows: using the face data set constructed in step Sa1 to train the improved YOLOv8 algorithm model, learn the features of the face, and accurately recognize the face; The database established in step Sa4 is a database that needs to store the recognized faces; Step Sa5 is specifically as follows: using VGGFace2 or CASIA-WebFace large face datasets to train the FaceNet algorithm model; Step Sa6 is as follows: deploy the built and trained YOLOv8 algorithm model to the edge computing platform, and use TensorRT acceleration technology and INT8 shaping quantization technology to accelerate the running speed of the face recognition algorithm on the edge computing platform; Step Sa7 is specifically as follows: using the YOLOv8 algorithm model to detect the face, then cropping the local face part for processing, and inputting the processed face image into the FaceNet network for further feature extraction and spatial mapping; Step Sa8 is specifically as follows: performing Euclidean distance calculation between the feature vector mapped in step Sa6 and the face vector in the database to determine the final face category.

7. The method for using the electronic sentinel system based on array radar, infrared and optics according to claim 6, characterized in that: In step S4, the specific process of the infrared and optical fusion algorithm is as follows: According to the conversion relationship of the coordinate system, the conversion from the world coordinate system to the pixel coordinate system is as follows: （13）； in, , , is the three-dimensional coordinates of the x-axis, y-axis, and z-axis of a point in the world coordinate system, is the depth of the target from the camera coordinate plane; make For the matrix , that is, the camera intrinsic parameter matrix; For the matrix , that is, the camera extrinsic matrix; Under the thermal infrared camera and optical camera, the conversion of the target world coordinates to pixel coordinates is shown in formulas (14) and (15) respectively: （14）； （15）； in, , They are the camera internal and external parameters of thermal infrared imaging, , They are the intrinsic and extrinsic parameters of the optical camera respectively; According to the basic rules of matrix operations and formulas (14) and (15), the pixel coordinates under the infrared camera and the pixel coordinates under the optical camera are converted to: （16）； Among them, the intrinsic and extrinsic parameters of the thermal infrared camera and optical camera are measured by Zhang Zhengyou's marking method.

Images