Method and system for detecting fatigue of nuclear power plant operators, electronic device and medium

Abstract

The embodiment of the application provides a kind of nuclear power plant operator's fatigue detection method and system, electronic equipment and medium, belong to nuclear power safety technical field.The method is applied to the fatigue detection device of nuclear power control room, comprising: obtaining the original image data that the image acquisition device of nuclear power control room is collected to operator, then the original image data is video frame disassembly, at least two continuous original video frames are obtained.Next, pixel color value calculation is carried out to each original video frame, reference color value is obtained, and all reference color values are signal construction according to the acquisition time sequence of original video frame, and the reflected light change signal is obtained.Then, according to the preset signal frequency band, the reflected light change signal is signal analyzed, the heart rate data and the breathing rate data of operator are obtained, and finally, according to the preset fatigue detection model, the heart rate data and the breathing rate data are fatigue detected.The reliability of fatigue detection result can be improved by the embodiment of the application.

Description

Technical Field

[0001] This application relates to the field of nuclear power safety technology, and in particular to a method and system for fatigue detection of nuclear power plant operators, electronic equipment and media. Background Technology

[0002] In the field of nuclear power safety technology, human resources are still needed at nuclear power plants to perform control operations, implement safety procedures, and continuously monitor the operational status of nuclear power system equipment to ensure the safe operation of nuclear power plants. Therefore, fatigue testing of operators in the control room of nuclear power plants helps to ensure the safety of nuclear power plants.

[0003] In related technologies, operators typically wear specialized hardware and sampling electrodes to collect physiological data (such as electroencephalograms, electrooculograms, and electrocardiograms), and then assess the operator's fatigue level based on this data. However, this data collection method causes significant discomfort to the operator, and prolonged electrode wear can lead to sweating, thus affecting the accuracy of the collected data and resulting in unreliable fatigue detection results. Therefore, improving the reliability of fatigue detection results has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a fatigue detection method, system, electronic equipment, and medium for nuclear power plant operators, aiming to improve the reliability of fatigue detection results.

[0005] To achieve the above objectives, a first aspect of this application proposes a fatigue detection method, applied to a fatigue detection device in a nuclear power plant control room, the method comprising:

[0006] Acquire raw image data of the operator collected by the image acquisition device in the nuclear power plant control room;

[0007] The original image data is decomposed into video frames to obtain at least two consecutive original video frames;

[0008] For each of the original video frames, the pixel color value is calculated to obtain the reference color value;

[0009] All the reference color values ​​are used to construct a signal according to the acquisition sequence of the original video frames to obtain the reflected light change signal;

[0010] The reflected light change signal is analyzed according to a preset signal frequency band to obtain the operator's heart rate and respiratory rate data.

[0011] Fatigue detection is performed on the heart rate data and respiratory rate data according to a preset fatigue detection model.

[0012] In some embodiments, calculating the pixel color value for each of the original video frames to obtain a reference color value includes:

[0013] The original video frame is used to extract the target region.

[0014] Extract the color value of each pixel in the target region to obtain the pixel color value;

[0015] The reference color value is obtained by averaging all the pixel color values.

[0016] In some embodiments, the target area includes a first target sub-region and a second target sub-region, wherein the first target sub-region is a partial area of ​​the operator's face, and the second target sub-region is a partial area of ​​the operator's chest; the step of constructing a signal from all the reference color values ​​according to the acquisition sequence of the original video frames to obtain the reflected light change signal includes:

[0017] Based on the acquisition timing sequence, a timing signal is generated for the reference color value corresponding to the first target sub-region to obtain the facial reflective light change signal;

[0018] Based on the acquisition timing sequence, a timing signal is generated for the reference color value corresponding to the second target sub-region to obtain the chest reflection light change signal;

[0019] The reflected light change signal is obtained based on the facial reflected light change signal and the chest reflected light change signal.

[0020] In some embodiments, the preset signal frequency band includes a first signal sub-frequency band and a second signal sub-frequency band; the step of performing signal analysis on the reflected light change signal according to the preset signal frequency band to obtain the operator's heart rate data and respiratory rate data includes:

[0021] Based on the first signal sub-frequency band, the facial reflective light change signal is extracted to obtain candidate reflective light change signals;

[0022] The candidate reflected light change signal is decomposed according to the preset decomposition constraints to obtain at least two candidate reflected photon signals;

[0023] The heart rate data is obtained by performing frequency domain analysis on the candidate reflected photon signals;

[0024] The respiratory rate data is obtained by performing frequency domain analysis on the chest reflectance change signal based on the second signal sub-frequency band.

[0025] In some embodiments, the step of performing frequency domain analysis based on the candidate reflected photon signals to obtain the heart rate data includes:

[0026] Each candidate reflected photon signal is subjected to a dimension reduction transformation to obtain a reconstructed signal;

[0027] The reconstructed signal is subjected to time-domain characteristic analysis to obtain the pulse time-domain signal;

[0028] The pulse time-domain signal is transformed into a frequency-domain signal to obtain the pulse frequency-domain signal.

[0029] The heart rate data is determined based on the pulse frequency domain signal.

[0030] In some embodiments, a plurality of image acquisition devices are arranged in the nuclear power plant control room; acquiring the raw image data collected by the operators by the image acquisition devices in the nuclear power plant control room includes:

[0031] Determine the current acquisition device from among multiple image acquisition devices;

[0032] Facial key points are identified from the initial image data acquired by the current acquisition device to obtain the facial key points in the image.

[0033] Based on the facial key points in the image, determine the acquisition object information that matches the initial image data;

[0034] In response to the information of the object being acquired indicating that the operator is not facing the current acquisition device, the current acquisition device is re-determined from multiple image acquisition devices;

[0035] In response to the information about the object being acquired reflecting that the operator is facing the current acquisition device, the initial image data acquired by the current acquisition device is determined as the original image data.

[0036] In some embodiments, before performing fatigue detection on the heart rate data and the respiratory rate data according to a preset fatigue detection model, the method further includes:

[0037] Acquire sitting pressure data collected by the pressure sensing device in the nuclear power plant control room; wherein, the sitting pressure data is the pressure applied by the operator to the pressure sensing device;

[0038] Acquire body temperature data of the operator collected by the body temperature measurement device in the nuclear power plant control room;

[0039] The step of performing fatigue detection on the heart rate data and respiratory rate data according to a preset fatigue detection model further includes:

[0040] Fatigue detection is performed on the heart rate data, respiratory rate data, sitting pressure data, and body temperature data based on the fatigue detection model.

[0041] To achieve the above objectives, a second aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0042] To achieve the above objectives, a third aspect of this application provides a fatigue detection system, the fatigue detection system comprising:

[0043] Fatigue detection device; wherein the fatigue detection device includes the electronic equipment described in the second aspect above;

[0044] Image acquisition device, used to acquire image data of operators in the nuclear power plant control room;

[0045] A pressure sensing device is used to acquire the operator's sitting posture pressure data;

[0046] A body temperature measuring device is used to measure the operator's body temperature data;

[0047] The pressure sensing device, the image acquisition device, and the body temperature measurement device are respectively connected to the fatigue detection device.

[0048] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0049] The fatigue detection method, system, electronic equipment, and media for nuclear power plant operators proposed in this application firstly acquire raw image data using image acquisition devices deployed in the nuclear power plant control room. This eliminates the need for operators to wear specialized hardware and sampling electrodes, avoiding discomfort for operators and the impact of sweating from prolonged electrode wear on data accuracy, thus improving reliability from the data acquisition source. Next, the raw image data is decomposed into video frames, and the color values ​​of the pixels are calculated and processed to convert the image data into reflected light change signals. Then, heart rate and respiratory rate data are analyzed according to a preset signal frequency band. This non-contact data acquisition method has minimal interference with the operator's normal work and can continuously monitor during operation. Finally, based on this accurately acquired physiological data, a preset fatigue detection model is used for fatigue detection, enabling a more objective and scientific assessment of the operator's fatigue level. Compared with traditional methods, this scheme does not rely on the operator's subjective feelings but analyzes based on actually acquired physiological data, without affecting the operator's physical sensations, greatly improving the reliability of fatigue detection results. Attached Figure Description

[0050] Figure 1 This is a flowchart of a fatigue detection method for nuclear power plant operators provided in an embodiment of this application;

[0051] Figure 2 This application provides a schematic diagram of a fatigue detection system for nuclear power plant operators in an embodiment.

[0052] Figure 3 yes Figure 1 The flowchart of step S101 in the text;

[0053] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0054] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0055] Figure 6 yes Figure 5 The flowchart of step S503 in the process;

[0056] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] First, let's analyze some of the terms used in this application:

[0061] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0062] Region of Interest (ROI): In machine vision and image processing, the area to be processed is delineated from the image using shapes such as rectangles, circles, ellipses, and irregular polygons.

[0063] Multi-view Canonical Correlation Analysis (MCCA) is an extension of classic Canonical Correlation Analysis (CCA) used to analyze the correlation between multiple sets of variables. Its goal is to project high-dimensional data from different views into a common low-dimensional subspace through linear transformation, maximizing the correlation of the projected data within this subspace. Assuming the existence of multiple data views, each represented by a high-dimensional set of variables, such as measurement data from different sensors, features from different modalities (e.g., images and text), or multiple decompositions of the same signal, MCCA finds a common representation space through linear transformation of these variable sets, where the projections of different views exhibit the strongest statistical correlation. In other words, MCCA extracts the common information from these multi-view data while removing noise or unique features specific to a particular view.

[0064] In the field of nuclear power safety technology, human resources are still needed at nuclear power plants to perform control operations, implement safety procedures, and continuously monitor the operational status of nuclear power system equipment to ensure the safe operation of nuclear power plants. Therefore, fatigue testing of operators in the control room of nuclear power plants helps to ensure the safety of nuclear power plants.

[0065] In related technologies, operators typically wear specialized hardware and sampling electrodes to collect physiological data (such as electroencephalograms, electrooculograms, and electrocardiograms), and then assess the operator's fatigue level based on this data. However, this data collection method causes significant discomfort to the operator, and prolonged electrode wear can lead to sweating, thus affecting the accuracy of the collected data and resulting in unreliable fatigue detection results. Therefore, improving the reliability of fatigue detection results has become an urgent technical problem to be solved.

[0066] Based on this, embodiments of this application provide a fatigue detection method and system, electronic equipment and medium for nuclear power plant operators, aiming to improve the reliability of fatigue detection results.

[0067] The fatigue detection method, system, electronic equipment, and medium for nuclear power plant operators provided in this application are specifically described through the following embodiments. First, the fatigue detection method for nuclear power plant operators in this application embodiment is described.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The fatigue detection method provided in this application relates to the field of nuclear power safety technology. The fatigue detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the fatigue detection method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0073] Figure 1 This is an optional flowchart of the fatigue detection method provided in the embodiments of this application. Figure 1 The method described herein, when applied to a fatigue detection device in a nuclear power plant control room, may include, but is not limited to, steps S101 to S106.

[0074] Step S101: Obtain raw image data collected by the operator from the image acquisition device in the nuclear power plant control room.

[0075] Step S102: Decompose the original image data into video frames to obtain at least two consecutive original video frames.

[0076] Step S103: Calculate the pixel color value for each original video frame to obtain the reference color value.

[0077] Step S104: Construct a signal from all reference color values ​​according to the acquisition sequence of the original video frames to obtain the reflected light change signal.

[0078] Step S105: Analyze the reflected light change signal according to the preset signal frequency band to obtain the operator's heart rate and respiratory rate data.

[0079] Step S106: Perform fatigue detection on heart rate and respiratory rate data according to the preset fatigue detection model.

[0080] Steps S101 to S106 of this embodiment first acquire raw image data using an image acquisition device deployed in the nuclear power plant control room. This eliminates the need for operators to wear dedicated hardware and sampling electrodes, avoiding discomfort for operators and the impact of sweating from prolonged electrode wear on data accuracy, thus improving reliability from the data acquisition source. Next, the raw image data is decomposed into video frames, and the color values ​​of the pixels are calculated to convert the image data into reflected light change signals. Then, heart rate and respiratory rate data are analyzed according to a preset signal frequency band. This non-contact data acquisition method has minimal interference with the operator's normal work and can continuously monitor during operation. Then, based on this accurately acquired physiological data, a preset fatigue detection model is used for fatigue detection, enabling a more objective and scientific assessment of the operator's fatigue level. Compared with traditional methods, this solution does not rely on the operator's subjective feelings but analyzes based on actually acquired physiological data, without affecting the operator's physical sensations, greatly improving the reliability of fatigue detection results.

[0081] It should be noted that the fatigue detection method provided in this application relies on the relevant hardware equipment system in the control room. Therefore, before explaining the steps, the fatigue detection system provided in this application will be introduced first. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of a fatigue detection system provided in this application. The fatigue detection system includes a fatigue detection device, an image acquisition device, a pressure sensing device, and a body temperature measurement device. The pressure sensing device, image acquisition device, and body temperature measurement device are respectively connected to the fatigue detection device. It should be noted that the connection between the two devices includes both data communication and wired connection; this embodiment does not strictly limit this.

[0082] In some embodiments, the image acquisition device acquires image data from operators in the nuclear power plant control room. The image acquisition device can be a camera, depth camera, or other acquisition device capable of capturing RGB images. In the nuclear power plant workstation scenario of this embodiment, multiple control terminals are arranged and deployed in the workstation control room for operators to operate. Each control terminal is equipped with an interactive screen and an image acquisition device.

[0083] In some embodiments, the pressure sensing device is used to acquire the operator's sitting posture pressure data. The pressure sensing device can be housed within a flexible seat cushion and can consist of an array of sensing modules, a matrix switch array, a signal processing module, a Bluetooth communication module, and a power supply module. Exemplarily, the sensing module is a flexible thin-film pressure sensing unit matrix with a sensing area designed to be 40×40 square centimeters. The sensing module is concealed within a latex pad of equal area, fitting well with the operator's seat. The matrix switch array is used to select the individual pressure sensing unit to be sampled each time. The signal processing module mainly consists of signal conditioning and sampling circuitry for the pressure sensing matrix, designed with a sampling frequency of 10Hz. The output data is a time series of the pressure sensing unit sampling matrix, stored in vector form. The Bluetooth communication module is responsible for transmitting the real-time sampling signals from the pressure sensing modules to the fatigue detection device. The power supply module is responsible for supplying power to the remaining modules. All components can be concealed behind the seat cushion and seat panel, creating data acquisition conditions that are imperceptible to the subject. Based on the acquired pressure matrix sequence data, the human sitting posture pressure distribution, i.e., sitting posture pressure data, is plotted. Four basic features related to fatigue were extracted from the pressure distribution obtained from the flexible pressure-sensing seat cushion: the x and y coordinates of the left and right ischial tuberosities. By analyzing the temporal characteristics of the basic features, their effective values ​​and the energy of the first frequency node of the third level of the wavelet packet were calculated. Finally, a fatigue feature vector composed of multiple features was constructed.

[0084] When the operator is working at their workstation, the pressure sensor can detect the pressure applied by the operator, which is the sitting pressure data. The sitting pressure data is then transmitted to the fatigue detection device, which uses this data to determine the changes in the operator's sitting posture (sitting upright, leaning to the left, leaning to the right, leaning forward, leaning backward, leaning forward to the left, leaning forward to the right, leaning backward to the left, leaning backward to the right, and vertical shaking) to further determine the operator's fatigue level.

[0085] In some embodiments, the body temperature measuring device is used to measure the operator's body temperature data, preferably a non-contact measuring device, such as an infrared thermometer. The body temperature measuring device can be integrated into a small wearable device, such as a smart bracelet or smart earphone, to monitor the operator's body temperature data in real time without causing discomfort to the operator during work. In other embodiments, these small wearable devices may also incorporate a positioning system, a blood oxygen saturation measuring device, and a heart rate measuring device, enabling simultaneous measurement of multiple data points from the operator.

[0086] In some embodiments, the fatigue detection device is used to perform fatigue detection on the operator based on data transmitted from the image acquisition device, pressure sensing device, and body temperature measuring device described above. The fatigue detection device includes electronic equipment for performing the fatigue detection method provided in the embodiments of this application. The electronic equipment will be further described in the following embodiments and will not be repeated here.

[0087] In step S101 of some embodiments, the raw image data is video images of the operator captured by an image acquisition device. In some nuclear power plant control rooms, multiple control terminals are deployed, meaning that the operator may face different control panels during operation. Since subsequent image processing requires recognizing the operator's facial features and other frontal physiological characteristics, if the image acquisition device only has one acquisition angle, some images in the final raw image data may not capture the operator's frontal view, resulting in missing relevant physiological data. To address this, multiple image acquisition devices can be deployed in the control room, each corresponding to one control terminal (e.g., [missing information - likely a specific control terminal]). Figure 2 (As shown). For the steps to obtain the raw image data, please refer to [link / reference needed]. Figure 3 Step S101 may include, but is not limited to, steps S301 to S305:

[0088] Step S301: Determine the current acquisition device from multiple image acquisition devices.

[0089] Step S302: Perform facial key point recognition on the initial image data acquired by the current acquisition device to obtain the facial key points in the image.

[0090] Step S303: Determine the acquisition object information that matches the initial image data based on the facial key points in the image.

[0091] In step S304, in response to the information of the object being acquired indicating that the operator is not facing the current acquisition device, the current acquisition device is re-determined from among multiple image acquisition devices.

[0092] Step S305: In response to the information of the object being acquired reflecting that the operator is facing the current acquisition device, the initial image data acquired by the current acquisition device is determined as the original image data.

[0093] In step S301 of some embodiments, one of the multiple image acquisition devices that can capture an image of the operator's front is selected as the current acquisition device. At the start of the workstation, a camera can be selected as the initial current acquisition device based on a pre-set priority order or position order. This default selection can be determined based on common working scenarios in the control room and the operator's typical operating posture. For example, if the most important processing tasks in the control room are displayed on the control terminal in the middle position, then the image acquisition device in the middle position can be defaulted as the initial current acquisition device during initialization to improve the success rate and efficiency of initial acquisition.

[0094] In step S302 of some embodiments, the initial image data is the image data captured by the acquisition device of the operator. Facial key point recognition can be achieved based on facial landmark detection algorithms such as Dlib or MediaPipe, where facial key points refer to the facial landmark data of the operator in the initial image data.

[0095] In step S303 of some embodiments, the information of the object to be collected may include whether the object corresponding to the initial image data is the operator himself, and whether the operator is facing the current acquisition device in the initial image data. Based on the identified facial key points in the image, the facial features and relative positions of the facial features of the object can be determined. Using this geometric feature information, combined with a pre-established operator facial feature database or model, it is determined whether the object corresponding to the currently acquired image data is the operator himself. The operator facial feature database stores facial feature templates or model parameters for each operator. These templates or parameters can be constructed based on features such as facial key point distribution and facial contour shape. The facial geometric features calculated in the current image are compared and matched with the templates in the database to calculate a similarity score. If the similarity score exceeds a set threshold, the object is considered to be the operator himself, and related personal information, such as name and employee number, can be obtained as object information.

[0096] Simultaneously, the facial features described above are used to determine whether the subject is facing the current acquisition device. In this embodiment, when the forehead, tip of the nose, and left and right cheeks are simultaneously identified in the initial image data, it can be determined that the subject is currently facing the current acquisition device.

[0097] In step S304 of some embodiments, if the forehead, nose tip, and left and right cheeks are not detected in the initial image data for a predetermined time period, it can be determined that the operator is not facing the current acquisition device, wherein the predetermined time period may be 20 seconds.

[0098] When the information of the object being acquired indicates that the operator is not facing the current acquisition device, the next image acquisition device can be selected as the current acquisition device in the order of the image acquisition device numbers, until the acquired image shows that the operator is currently facing the corresponding image acquisition device, and that image acquisition device is then set as the latest current acquisition device.

[0099] In some embodiments, if no facial key points of the operator are identified in the images captured by all image acquisition devices, it can be determined that the operator is not working at this time. In this case, all image acquisition devices can be controlled to enter standby mode to avoid frequent switching of cameras.

[0100] In step S305 of some embodiments, to facilitate database retrieval of video files, the original image data is saved in segments by hour. In some embodiments, to ensure the continuity of the original image data, whenever an image acquisition device is identified as the current acquisition device, the acquired initial image data is used as the original image data.

[0101] Steps S301 to S305 as illustrated in this embodiment of the application, by rationally distributing and installing multiple image acquisition devices, can comprehensively cover the operator's working area, ensuring that there is suitable equipment for image acquisition in any scenario, thereby improving the comprehensiveness and reliability of data acquisition. Facial key point recognition of the initial image data can accurately obtain the operator's facial feature point data, accurately identifying whether the subject being acquired is the operator. Simultaneously, by using specific facial features, it can determine whether the operator is facing the acquisition device, achieving accurate identification of the subject's identity and posture. When it is found that the operator is not facing the current acquisition device, the image acquisition devices are switched sequentially according to their numerical order, ensuring that the final obtained raw image data includes a highly complete facial image of the operator, thereby improving the reliability of the raw image data.

[0102] In step S102 of some embodiments, professional video processing software or algorithm libraries, such as OpenCV, are used to perform video frame decomposition on the acquired raw image data, and the final video frame obtained is the original video frame. During the decomposition process, it is ensured that consecutive raw video frames are accurately extracted according to the camera's frame rate to avoid frame loss or misalignment. For each frame, its pixel information, image format, timestamp, and other key data are fully preserved for accurate subsequent analysis and processing.

[0103] In step S103 of some embodiments, please refer to Figure 4 In some embodiments, step S103 may include, but is not limited to, steps S401 to S403:

[0104] Step S401: Extract regions from the original video frames to obtain the target region.

[0105] Step S402: Extract the color value of each pixel in the target area to obtain the pixel color value.

[0106] Step S403: Calculate the average value of all pixel color values ​​to obtain the reference color value.

[0107] In step S401 of some embodiments, the target region includes a first target sub-region and a second target sub-region. The first target sub-region is a partial area of ​​the operator's face, and the second target sub-region is a partial area of ​​the operator's chest. In this embodiment, the second target sub-region is the area corresponding to the operator's heart. Specifically, the first target sub-region may include a partial area of ​​the forehead, a partial area of ​​the left cheek, and a partial area of ​​the right cheek. It is understood that selecting multiple sub-regions as the target region has the following advantages in subsequent image processing: 1. When the face is partially occluded (e.g., hair covers the face when the head is lowered, or the operator's palm covers the face when the hand is supporting the face), selecting multiple first target sub-regions can make full use of the unoccluded area, thereby avoiding data loss. 2) Significant tilting and shaking of the face may cause some target sub-regions to be out of the image, resulting in corresponding data loss. 3) Due to interference from factors such as the angle of the light source and changes in illumination, target regions of different positions and sizes on the face are subject to different interferences, and the signal quality is also different. However, these target regions can be considered to contain heartbeat information of the same origin but different intensities, which lays the foundation for obtaining the correct heart rate information through subsequent signal decomposition.

[0108] For the extraction of target regions, operators can directly select the region of interest through the input device, or it can be determined by a pre-trained target recognition model. This method can adjust the range of the target region in real time according to the positional changes of facial key points to ensure that the effective region related to physiological signals can always be accurately extracted.

[0109] In step S402 of some embodiments, the pixel color value is the RGB value of the current pixel.

[0110] In step S403 of some embodiments, the reference color value is the average value of all pixels within the target area across the three channels of the RGB color space. The reference color value satisfies the following analytical expression:

[0111]

[0112] Where m represents the sequence of original video frames, and k represents the sequence of the target sub-region. S k (m) represents the reference color value. ROI k (x,y,m) represents the pixel color value at pixel coordinates (x,y) in the k-th target sub-region. R(m), G(m), and B(m) represent the color values ​​of the red, green, and blue channels in the m-th original video frame. PN k This represents the total number of pixels in the k-th target sub-region.

[0113] It should be noted that when the target area is composed of multiple sub-regions, a reference color value is calculated separately for each sub-region, which is the average pixel color value of the sub-region.

[0114] Steps S401 to S403, as illustrated in this embodiment, extract the target region from the original video frame. This allows focus on key areas closely related to the operator's physiological signals, facilitating more accurate capture of color features related to physiological changes and laying the foundation for obtaining effective physiological signal data. Color values ​​are extracted from each pixel in the target region, and a reference color value is calculated by averaging all pixel color values. This effectively integrates color information within the region, reducing the impact of individual pixel noise and local color changes, resulting in a feature value that comprehensively reflects the overall color change trend of the target region. This reference color value can serve as a stable and reliable indicator in subsequent signal analysis, improving the accuracy and stability of heart rate and respiratory rate data analysis.

[0115] In step S104 of some embodiments, when the target area is composed of a first target sub-region and a second target sub-region, a timing signal is generated based on the acquisition timing of the original video frames for the reference color values ​​corresponding to the first target sub-region, resulting in a facial reflective light change signal. Simultaneously, a timing signal is generated based on the acquisition timing for the reference color values ​​corresponding to the second target sub-region, resulting in a chest reflective light change signal. It should be noted that the facial reflective light change signal and the chest reflective light change signal can be Remote Photo Plethysmo Graph (rPPG) signals. The principle of rPPG signals is based on the camera capturing the slight color changes caused by blood flowing across the facial skin. Because blood flow is affected by heart activity, analyzing the fluctuation frequency of this color change over time can reflect heart rate information. It should be noted that an rPPG signal is constructed for each target sub-region. Then, the reflective light change signal corresponding to the target area is obtained based on the facial reflective light change signal and the chest reflective light change signal.

[0116] Please see Figure 5 In some embodiments, step S105 includes a first signal sub-band and a second signal sub-band. Specifically, the first signal sub-band may be [0.7Hz, 4Hz], and the second signal sub-band may be [0.1Hz, 0.4Hz]. Step S105 may include, but is not limited to, steps S501 to S504.

[0117] Step S501: Extract the facial reflective light change signal based on the first signal sub-frequency band to obtain candidate reflective light change signals.

[0118] Step S502: Decompose the candidate reflected light change signal according to the preset decomposition constraints to obtain at least two candidate reflected photon signals.

[0119] Step S503: Perform frequency domain analysis based on the candidate reflected photon signals to obtain heart rate data.

[0120] Step S504: Perform frequency domain analysis on the chest reflection light change signal based on the second signal sub-frequency band to obtain respiratory rate data.

[0121] In step S501 of some embodiments, the candidate reflected light change signal is the facial reflected light change signal with a frequency in the first signal sub-band.

[0122] In step S502 of some embodiments, to fully extract information more closely related to the heartbeat from the signal, the variational mode extraction (VME) algorithm can be used to decompose the candidate reflected light change signal. However, if the candidate reflected light change signal is over-decomposed, it will not only increase the computational load but may also introduce excessive noise interference. Conversely, if the signal decomposition is insufficient, it will be impossible to fully extract information related to the heartbeat, thereby reducing the accuracy of subsequent signal analysis. Therefore, it is necessary to determine the termination condition of the signal decomposition, that is, the decomposition constraint condition. The signal after the candidate reflected light change signal is decomposed is the candidate reflected photon signal.

[0123] Specifically, it is necessary to determine the significance of the dominant peak of the candidate reflected light variation signal within the range of [0.7, 4Hz]. This can be measured by the ratio between the amplitude of the dominant peak of the candidate reflected light variation signal and the average amplitude of the first signal sub-band. When the significance of the dominant peak is greater than or equal to a preset threshold, it indicates that the quality of the candidate reflected light variation signal is good, and excessive decomposition is unnecessary. In this case, the decomposition terminates under the following condition: Where Er represents the energy of the residual signal after decomposition, and Es represents the total energy of the candidate reflected light variation signal. If the main peak value is significantly less than a preset threshold, it indicates that the quality of the candidate reflected light variation signal is poor and contains a lot of noise, requiring thorough decomposition. The corresponding decomposition termination condition is...

[0124] In step S503 of some embodiments, please refer to Figure 6 Step S503 includes, but is not limited to, steps S601 to S604:

[0125] Step S601: Perform a dimension reduction transformation on each candidate reflected photon signal to obtain the reconstructed signal.

[0126] Step S602: Perform time-domain characteristic analysis on the reconstructed signal to obtain the pulse time-domain signal.

[0127] Step S603: Perform frequency domain transformation on the pulse time domain signal to obtain the pulse frequency domain signal.

[0128] Step S604: Determine heart rate data based on the pulse frequency domain signal.

[0129] In step S601 of some embodiments, the candidate reflector signal can be reduced in dimension and its features extracted by multi-view canonical correlation analysis (MCCA) to output multiple signals containing real pulse information, which is the reconstructed signal.

[0130] In step S602 of some embodiments, time-domain characteristic analysis is performed on the reconstructed signal. The time-domain characteristics may include the peak-to-peak time interval, the rate of change of the signal, and the average amplitude of the signal. These characteristics can evaluate the quality of the reconstructed signal. For example, time-series analysis is performed on each reconstructed signal, and the reconstructed signal with the lowest rate of change is selected as the pulse time-domain signal.

[0131] In step S603 of some embodiments, a fast Fourier transform can be performed on the pulse time-domain signal to obtain the pulse frequency-domain signal.

[0132] In step S604 of some embodiments, the maximum peak value of the pulse frequency domain signal in the frequency domain is the frequency corresponding to the heartbeat, and the operator's heart rate data can be calculated according to the following analytical expression:

[0133] HR = f hr ×60 (2),

[0134] Where HR represents heart rate data, f hr This represents the maximum peak value of the pulse frequency domain signal.

[0135] In other embodiments, the time interval between heartbeats (RR interval) can be obtained from the peak value of the pulse frequency domain signal. Subsequently, during fatigue detection in the model, heart rate variability (HRV) feature analysis can be performed based on this interval to obtain multiple feature information, as summarized in Table 1, thereby further improving the reliability of the fatigue detection results.

[0136]

[0137] Table 1

[0138] Steps S601 to S604, as illustrated in the embodiments of this application, preserve key information by performing a dimensionality reduction transformation on the candidate reflected photon signal, reducing data processing volume and computational complexity, and improving the efficiency of subsequent analysis. This helps remove redundant information and noise interference, making the reconstructed signal more prominent in reflecting the key components of pulse characteristics, laying a good foundation for accurate pulse signal analysis. Time-domain characteristic analysis of the reconstructed signal allows for in-depth exploration of the pulse signal's variation patterns over time. Converting the pulse time-domain signal to a frequency-domain signal and using a Fast Fourier Transform reveals the signal's spectral distribution in the frequency domain, finally accurately identifying the frequency peak corresponding to the heart rate, providing a direct basis for determining heart rate data.

[0139] In step S504 of some embodiments, the portion of the signal with the frequency band of the second signal sub-band (i.e., [0.1Hz, 0.4Hz]) is extracted from the signal of the change in chest reflective light, and a fast Fourier transform is performed on this portion of the signal. The frequency corresponding to the maximum peak value in the obtained frequency distribution is the respiratory rate. The respiratory rate can be calculated according to the following analytical formula:

[0140] RR = f rr ×60 (3),

[0141] Where RR represents the number of breaths per minute, i.e., the respiratory rate. rr Indicates respiratory rate.

[0142] Steps S501 to S504, as illustrated in this embodiment, accurately extract the facial reflective light change signal based on the first signal sub-frequency band, effectively focusing on the signal components related to heart rate and eliminating interference from other irrelevant frequency signals. By rationally decomposing the candidate reflective light change signal according to preset decomposition constraints, the complex reflective light change signal can be decomposed into multiple more characteristic candidate reflective photon signals, facilitating in-depth mining of heart rate information within the signal. Frequency domain analysis is performed based on the candidate reflective photon signals, accurately determining heart rate data using frequency domain characteristics. Simultaneously, frequency domain analysis is performed on the chest reflective light change signal based on the second signal sub-frequency band to obtain respiratory rate data, achieving non-contact detection of this important physiological indicator of respiration. Combined with facial heart rate detection, this forms a multimodal physiological signal detection system.

[0143] In other embodiments, other modal data, such as the operator's eye closure state, head posture, and predicted gaze area, are also detected on the raw image data.

[0144] For example, the operator's eye-closing state can be achieved as follows: Using the top-left corner of the sample image as the origin, the horizontal axis as the x-axis, and the vertical axis as the y-axis, a training set can be pre-constructed, collecting multiple sample images and extracting facial key points from these images. Based on the portrait in the sample image, the left corner of the left eye and the right corner of the right eye are determined, and the coordinates of the rotation center point, a(ax, ay), are calculated based on the coordinates of these two key points, as shown in the following analytical expression:

[0145]

[0146] Wherein, P37.x and P37.y represent the x and y coordinates of the key point numbered at the left corner of the eye, respectively. P46.x and P46.y represent the x and y coordinates of the key point at the right corner of the eye, respectively. All sample images are aligned according to the coordinates of the rotation center points mentioned above and labeled. In this embodiment, the labels are open eyes and closed eyes. Then, a neural network-based eye closure detection model can be constructed and trained using a pre-built labeled open / closed eye detection dataset. Finally, the trained eye closure detection model is used to detect the original video frames to determine the operator's eye closure state. For example, the original video frame is processed to a size of 3×32×32 and used as the input of the neural network. Six convolutional kernels are used to perform convolution operations on the input sample images. The convolutional kernel size is 5×5, the stride is 1, and no edge padding is performed, resulting in a feature vector of size 6×28×28. Max pooling is performed on the feature vector with a kernel size of 2×2, a stride of 2, and no padding, resulting in a feature vector size of 6×14×14. Convolution is then performed using 16 kernels with a kernel size of 5×5, a stride of 1, and no padding, resulting in a feature vector size of 16×10×10. Max pooling is then performed on the feature vector with a kernel size of 2×2, a stride of 2, and no padding, resulting in a feature vector size of 16×5×5. Finally, convolution is performed using 120 kernels with a kernel size of 5×5, a stride of 1, and no padding, resulting in a feature vector size of 120×1×1. The obtained feature vector is input into the first fully connected layer for fully connected computation. The first fully connected layer has 120 neurons, resulting in a feature vector of size 1×120. The feature vector is then input into the second fully connected layer for fully connected computation. The second fully connected layer has 2 neurons, corresponding to the two category labels, resulting in a feature vector of size 1×2. The softmax function is used to normalize the feature vector to obtain the probability that the input eye image is open or closed.

[0147] For example, head pose can be acquired in the following way. In the above embodiment, facial key points and their coordinates have been acquired. Based on these key point coordinates, combined with a general three-dimensional coordinate model of head key points, the rotation matrix of the head is calculated using an N-point perspective pose solving algorithm. The rotation matrix describes the rotation state of the head relative to the camera. Then, the head pose is represented as pitch, yaw, and roll in a spatial coordinate system using this matrix, thus completely characterizing the three-dimensional pose of the head. In the above calculation process, the calibration of the image acquisition device is a key step. The intrinsic parameter matrix and distortion parameters of the image acquisition device can be calibrated using the Zhang Zhengyou calibration method to complete the coordinate transformation relationship between the world coordinate system and the camera coordinate system. According to the calibration results, the facial key points are aligned with a standard face model, and the head rotation matrix and head translation vector are calculated. Finally, the head pose can be clearly described.

[0148] For example, the acquisition of the predicted gaze region can be achieved as follows: The public dataset DDGC-DB1 is used as the training set for the training model. The size of each sample in the training set is uniformly adjusted to 224×224. A facial image is cropped from the scaled image, with the width denoted as L1 and the height as H1. The images of both eyes are cropped, centered on the geometric center of the eye region, with the distance between the inner corners of the eyes denoted as L2 and the height as H2. Feature extraction is performed on the cropped facial and eye images using convolutional layers of a VGG16 neural network, resulting in three 1×4096 feature vectors, denoted as ξ, ψ, and γ, respectively. The weighted sum vector Γ is then calculated. The Euclidean distances o1 and o2 between vectors ξ and ψ are calculated as shown in the following analytical expression.

[0149]

[0150]

[0151] Γ=ξ+o1ψ+o2γ (8),

[0152] Where k = 1, 2, 3, ..., 4095. ξk represents the k-th element of vector ξ, ψk represents the k-th element of vector ψ, and γk represents the k-th element of vector γ.

[0153] After the vector Γ is calculated by two fully connected layers, the result is normalized using the softmax function to obtain the output vector R. The element values ​​in vector R are the predicted probability values ​​of each category, and the position of the element with the maximum value is the prediction result of the gaze region.

[0154] Before step S106 in some embodiments, seated pressure data collected by a pressure sensing device in the nuclear power plant control room is acquired. The seated pressure data is the pressure applied by the operator to the pressure sensing device, which has been explained in detail in the embodiments of the fatigue detection system and will not be repeated here.

[0155] In some embodiments, body temperature data collected by a body temperature measuring device in the nuclear power plant control room from the operator can also be acquired.

[0156] In step S106 of some embodiments, fatigue detection is performed on heart rate data and respiratory rate data according to a preset fatigue detection model to obtain the operator's fatigue state, which can be either fatigue or wakefulness.

[0157] In addition to detecting fatigue based on respiratory rate and heart rate data, the fatigue detection model can also arbitrarily combine the aforementioned sitting pressure data, body temperature data, head posture, eye closure status, predicted gaze area, and HRV features obtained from heart rate data as input. Corresponding weight parameters are pre-set for different modalities of data, and the model's prediction results and weight parameters are combined to comprehensively assess the operator's fatigue level.

[0158] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described fatigue detection method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0159] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0160] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0161] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the fatigue detection method of the embodiments of this application.

[0162] The input / output interface 703 is used to implement information input and output;

[0163] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0164] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0165] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described fatigue detection method.

[0167] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0168] The fatigue detection method, system, electronic equipment, and medium for nuclear power plant operators provided in this application first acquire raw image data using an image acquisition device deployed in the nuclear power plant control room. This eliminates the need for operators to wear specialized hardware and sampling electrodes, avoiding discomfort and the inaccuracy of data due to sweating from prolonged electrode use, thus improving reliability from the data acquisition source. Next, the raw image data is decomposed into video frames, and the color values ​​of the pixels are calculated to convert the image data into reflected light change signals. Then, heart rate and respiratory rate data are analyzed according to a preset signal frequency band. This non-contact data acquisition method has minimal interference with the operator's normal work and can continuously detect fatigue during operation. Finally, based on this accurately acquired physiological data, a preset fatigue detection model is used for fatigue detection, enabling a more objective and scientific assessment of the operator's fatigue level. Compared to traditional methods, this solution does not rely on the operator's subjective feelings but analyzes actual collected physiological data, without affecting the operator's physical sensations, greatly improving the reliability of fatigue detection results.

[0169] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0170] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0173] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0174] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for fatigue detection of nuclear power plant operators, characterized in that, Fatigue detection equipment used in nuclear power plant control rooms The method includes: Acquire raw image data of the operator collected by the image acquisition device in the nuclear power plant control room; The original image data is decomposed into video frames to obtain at least two consecutive original video frames; For each of the original video frames, the pixel color value is calculated to obtain the reference color value; All the reference color values ​​are used to construct a signal according to the acquisition sequence of the original video frames to obtain the reflected light. Change signal; The operator's heart rate data is obtained by analyzing the reflected light change signal according to a preset signal frequency band. and respiratory rate data; Acquire sitting pressure data collected by the pressure sensing device in the nuclear power plant control room; wherein, the sitting pressure data The pressure applied by the operator to the pressure sensing device; Acquire body temperature data of the operator collected by the body temperature measurement device in the nuclear power plant control room; Fatigue detection is performed on the heart rate data, respiratory rate data, sitting pressure data, and body temperature data according to a preset fatigue detection model, including: obtaining the ischial tuberosity coordinates of the operator based on the sitting pressure data; constructing a fatigue feature vector based on the ischial tuberosity coordinates; and performing fatigue detection on the heart rate data, respiratory rate data, fatigue feature vector, and body temperature data according to the fatigue detection model.

2. The fatigue detection method according to claim 1, characterized in that, The process of performing each of the original video frames Pixel color value calculation yields reference color values, including: The original video frame is used to extract the target region. Extract the color value of each pixel in the target region to obtain the pixel color value; The reference color value is obtained by averaging all the pixel color values.

3. The fatigue detection method according to claim 2, characterized in that, The target area includes a first target sub-region. The first target sub-region is a partial area of ​​the operator's face, and the second target sub-region is a second target sub-region. The sub-region is a local area of ​​the operator's chest; the reference color values ​​are then calculated according to the original video frames. The acquisition timing sequence is used to construct the signal, obtaining the reflected light change signal, including: Based on the acquisition timing, a timing signal is generated for the reference color value corresponding to the first target sub-region. Successfully obtained the signal of changes in facial reflected light; Based on the acquisition timing, a timing signal is generated for the reference color value corresponding to the second target sub-region. Successfully, the signal of change in chest reflective light was obtained; The reflected light change signal is obtained based on the facial reflected light change signal and the chest reflected light change signal.

4. The fatigue detection method according to claim 3, characterized in that, The preset signal frequency band includes a first signal. Sub-frequency band and second signal sub-frequency band; the step of performing signal analysis on the reflected light change signal according to the preset signal frequency band. The operator's heart rate and respiratory rate data are obtained, including: Based on the first signal sub-frequency band, the facial reflective light change signal is extracted to obtain candidate reflective light. Change signal; The candidate reflected light variation signal is decomposed according to preset decomposition constraints to obtain at least two candidate signals. Select the reflected photon signal; The heart rate data is obtained by performing frequency domain analysis on the candidate reflected photon signals; Based on the second signal sub-frequency band, frequency domain analysis was performed on the chest reflective light change signal to obtain the respiratory rate. Rate data.

5. The fatigue detection method according to claim 4, characterized in that, The based on the candidate reflected photon signal Frequency domain analysis was performed to obtain the heart rate data, including: Each candidate reflected photon signal is subjected to a dimension reduction transformation to obtain a reconstructed signal; The reconstructed signal is subjected to time-domain characteristic analysis to obtain the pulse time-domain signal; The pulse time-domain signal is transformed into a frequency-domain signal to obtain the pulse frequency-domain signal. The heart rate data is determined based on the pulse frequency domain signal.

6. The fatigue detection method according to any one of claims 1 to 5, characterized in that, The nuclear power control room is in the middle of the Multiple of the aforementioned image acquisition devices are listed; The image acquisition device for the nuclear power plant control room captures images from the operator. The initial image data includes: Determine the current acquisition device from among multiple image acquisition devices; Facial key point recognition is performed on the initial image data acquired by the current acquisition device to obtain the facial key points in the image. point; Based on the facial key points in the image, determine the acquisition object information that matches the initial image data; In response to the acquisition object information indicating that the operator is not facing the current acquisition device, acquisition is performed from multiple images. The current data acquisition device is redefined within the data acquisition device; In response to the information about the object being collected indicating that the operator is facing the current acquisition device, the current acquisition device is... The initial image data acquired is defined as the original image data.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing... A computer program, which, when executed by the processor, implements the fatigue detection method according to any one of claims 1 to 6. method.

8. A fatigue detection system, characterized in that, The fatigue detection system includes: Fatigue detection device; wherein the fatigue detection device includes the electronic device as described in claim 7; Image acquisition device, used to acquire image data of operators in the nuclear power plant control room; A pressure sensing device is used to acquire the operator's sitting posture pressure data; A body temperature measuring device is used to measure the operator's body temperature data; The pressure sensing device, the image acquisition device, and the body temperature measurement device are respectively connected to the fatigue detection device. Connect the measuring device.

9. A computer-readable storage medium storing a computer program, characterized in that... When the computer program is executed by the processor, it implements the fatigue detection method according to any one of claims 1 to 6.

Images