Fatigue detection method and system for nuclear power plant operator, electronic equipment and medium

Abstract

The embodiment of the invention provides a fatigue detection method and system for a nuclear power plant operator, electronic equipment and a medium, and belongs to the technical field of nuclear power safety. The method is applied to a fatigue detection device of a nuclear power control room, and comprises the following steps: acquiring original image data acquired by an image acquisition device of the nuclear power control room for an operator, and then performing video frame disassembly on the original image data to obtain at least two continuous original video frames. Next, pixel color value calculation is carried out on each original video frame to obtain reference color values, and signal construction is carried out on all the reference color values according to the acquisition time sequence of the original video frames to obtain reflected light change signals; signal analysis is carried out on the reflected light change signal according to a preset signal frequency band, heart rate data and respiration rate data of the operator are obtained, and finally fatigue detection is carried out on the heart rate data and the respiration rate data according to a preset fatigue detection model. According to the embodiment of the invention, the reliability of fatigue detection results can be improved.

Description

Technical Field

[0001] This application relates to the field of nuclear power safety technologies, and particularly to a fatigue detection method and system for nuclear power plant operators, an electronic device, and a medium. Background Art

[0002] In the field of nuclear power safety technologies, human labor is still required to perform control operations in nuclear power plants, execute safety regulations, and continuously observe the operation status of nuclear power system equipment to ensure the safe operation of nuclear power plants. Therefore, fatigue detection of operators in the control room of nuclear power plants helps to ensure the safety of nuclear power plants.

[0003] In related technologies, operators are usually required to wear special hardware devices and sampling electrodes to collect physiological data (such as electroencephalogram, electrooculogram, and electrocardiogram), and then evaluate the fatigue level of the operators based on the collected data. However, such a collection method will cause obvious discomfort to the operators, and sweating will occur after wearing the electrodes for a long time, which will affect the accuracy of the collected data, resulting in the lack of reliability of the fatigue detection results based on these data. Therefore, how to improve the reliability of fatigue detection results has become a technical problem to be solved urgently. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a fatigue detection method and system for nuclear power plant operators, an electronic device, and a medium, aiming to improve the reliability of fatigue detection results.

[0005] To achieve the above object, a first aspect of the embodiments of this application proposes a fatigue detection method, which is applied to a fatigue detection device in a nuclear power control room. The method includes:

[0006] Obtain the original image data collected by an image acquisition device in the nuclear power control room for the operator;

[0007] Perform video frame decomposition on the original image data to obtain at least two consecutive original video frames;

[0008] Calculate the pixel color values for each of the original video frames to obtain reference color values;

[0009] Construct a signal based on all the reference color values according to the acquisition timing sequence of the original video frames to obtain a reflected light change signal;

[0010] Perform signal analysis on the reflected light change signal according to a preset signal frequency band to obtain the heart rate data and respiration rate data of the operator;

[0011] Perform fatigue detection on the heart rate data and the respiration rate data according to a preset fatigue detection model.

[0012] In some embodiments, calculating the pixel color value of each of the original video frames to obtain a reference color value includes:

[0013] Performing region extraction on the original video frame to obtain a target region;

[0014] Extracting the color value of each pixel in the target region to obtain a pixel color value;

[0015] Calculating the average value based on all the pixel color values to obtain the reference color value.

[0016] In some embodiments, the target region includes a first target sub-region and a second target sub-region. The first target sub-region is a local region of the operator's face, and the second target sub-region is a local region of the operator's chest. Constructing a reflected light change signal according to the acquisition time sequence of all the reference color values includes:

[0017] Generating a time sequence signal based on the reference color value corresponding to the first target sub-region according to the acquisition time sequence to obtain a facial reflected light change signal;

[0018] Generating a time sequence signal based on the reference color value corresponding to the second target sub-region according to the acquisition time sequence to obtain a chest reflected light change signal;

[0019] Obtaining the reflected light change signal based on the facial reflected light change signal and the chest reflected light change signal.

[0020] In some embodiments, the preset signal frequency band includes a first signal sub-band and a second signal sub-band. Analyzing the reflected light change signal according to the preset signal frequency band to obtain the heart rate data and respiratory rate data of the operator includes:

[0021] Extracting a candidate reflected light change signal from the facial reflected light change signal according to the first signal sub-band;

[0022] Decomposing the candidate reflected light change signal according to a preset decomposition constraint condition to obtain at least two candidate reflected light sub-signals;

[0023] Performing frequency domain analysis on the candidate reflected light sub-signals to obtain the heart rate data;

[0024] Performing frequency domain analysis on the chest reflected light change signal according to the second signal sub-band to obtain the respiratory rate data.

[0025] In some embodiments, performing frequency domain analysis on the candidate reflected light sub-signals to obtain the heart rate data includes:

[0026] Perform dimensionality reduction transformation processing on each of the candidate reflected photon signals to obtain a reconstructed signal;

[0027] Perform time-domain characteristic analysis on the reconstructed signal to obtain a pulse time-domain signal;

[0028] Perform frequency-domain transformation on the pulse time-domain signal to obtain a pulse frequency-domain signal;

[0029] Determine the heart rate data according to the pulse frequency-domain signal.

[0030] In some embodiments, a plurality of the image acquisition devices are arranged in the nuclear power control room; obtaining the original image data collected by the image acquisition device in the nuclear power control room for the operator includes:

[0031] Determine the current acquisition device from the plurality of image acquisition devices;

[0032] Perform facial key point recognition on the initial image data collected by the current acquisition device to obtain the facial key points in the picture;

[0033] Determine the acquisition object information matching the initial image data according to the facial key points in the picture;

[0034] In response to the acquisition object information indicating that the operator is not facing the current acquisition device directly, re-determine the current acquisition device from the plurality of image acquisition devices;

[0035] In response to the acquisition object information indicating that the operator is facing the current acquisition device directly, determine the initial image data collected by the current acquisition device as the original image data.

[0036] In some embodiments, before performing fatigue detection on the heart rate data and the respiratory rate data according to a preset fatigue detection model, it further includes:

[0037] Obtain the sitting posture pressure data collected by the pressure sensing device in the nuclear power control room; wherein, the sitting posture pressure data is the pressure exerted by the operator on the pressure sensing device;

[0038] Obtain the body temperature data collected by the body temperature measuring device in the nuclear power control room for the operator;

[0039] Performing fatigue detection on the heart rate data and the respiratory rate data according to the preset fatigue detection model further includes:

[0040] Perform fatigue detection on the heart rate data, the respiratory rate data, the sitting posture pressure data, and the body temperature data according to the fatigue detection model.

[0041] To achieve the above object, a second aspect of the embodiments of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0042] To achieve the above object, a third aspect of the embodiments of the present application provides a fatigue detection system, which includes:

[0043] A fatigue detection device; wherein, the fatigue detection device includes the electronic device described in the second aspect above;

[0044] An image acquisition device, configured to acquire image data of an operator in a nuclear power control room;

[0045] A pressure sensing device, configured to obtain the sitting posture pressure data of the operator;

[0046] A body temperature measurement device, configured to measure the body temperature data of the operator;

[0047] Wherein, the pressure sensing device, the image acquisition device and the body temperature measurement device are respectively connected to the fatigue detection device.

[0048] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0049] The fatigue detection method and system, electronic device and medium for nuclear power plant operators proposed by the present application. First, by using the image acquisition device deployed in the nuclear power control room to obtain the original image data, there is no need for the operator to wear special hardware devices and sampling electrodes, avoiding the discomfort caused to the operator and the problem of affecting data accuracy due to sweating during long-term electrode wearing, and improving the reliability from the source of data acquisition. Then, the original image data is disassembled into video frames, and the color values of the pixels are calculated and processed to convert the image data into a reflected light change signal. Then, the heart rate data and respiratory rate data are parsed according to the preset signal frequency band. This non-contact data acquisition method has minimal interference with the normal work of the operator and can continuously detect during the work process. Then, based on these accurately acquired physiological data, the preset fatigue detection model is used for fatigue detection, which can more objectively and scientifically evaluate the fatigue degree of the operator. Compared with the traditional method, this solution does not rely on the subjective feelings of the operator, but analyzes based on the actually collected physiological data and does not affect the physical sensation of the operator, greatly improving the reliability of the fatigue detection result. Description of the Drawings

[0050] Figure 1 It is a flowchart of the fatigue detection method for nuclear power plant operators provided by an embodiment of the present application;

[0051] Figure 2 Schematic diagram of the fatigue detection system for nuclear power plant operators provided by an embodiment of the present application;

[0052] Figure 3 is Figure 1 The flowchart of step S101 in

[0053] Figure 4 is Figure 1 The flowchart of step S103 in

[0054] Figure 5 is Figure 1 The flowchart of step S105 in

[0055] Figure 6 is Figure 5 The flowchart of step S503 in

[0056] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0057] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from the module division in the device or the order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] First, several nouns involved in the present application are analyzed:

[0061] Artificial Intelligence (AI): A new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; AI is a branch of computer science. It attempts to understand the essence of intelligence and produce an intelligent machine that can respond in a way similar to human intelligence. Research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. AI can simulate the information process of human consciousness and thinking. It also refers to the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0062] Region of Interest (ROI): In machine vision and image processing, it is the area that needs to be processed outlined in the form of a square, circle, ellipse, irregular polygon, etc. from the image to be processed.

[0063] Multi-view Canonical Correlation Analysis (MCCA): An extension of classical Canonical Correlation Analysis (CCA) for analyzing the correlation between multiple sets of variables. Its goal is to project the high-dimensional data of different views into a common low-dimensional subspace through linear transformation, so that the projected data has the maximum correlation within this subspace. Suppose there are multiple data views, each view is represented by a set of high-dimensional variables, such as measurement data from different sensors, features of different modalities (such as images and texts), or multiple decomposition forms of the same signal. By performing linear transformation on these sets of variables, MCCA can find a common representation space, so that the projections of different views in this space have the strongest statistical correlation. In other words, what MCCA extracts is the common information in these multi-view data, while removing the noise or unique features related to a specific view.

[0064] In the field of nuclear power safety technology, it is still necessary for humans to carry out control operations in nuclear power plants, execute safety regulations, and continuously observe the operation status of nuclear power system equipment to ensure the safe operation of nuclear power plants. Therefore, fatigue detection of the operators in the control room of nuclear power plants helps to ensure the safety of nuclear power plants.

[0065] In the related art, an operator is usually made to wear special hardware devices and sampling electrodes to collect physiological data (such as electroencephalogram, electrooculogram, and electrocardiogram), and then the fatigue level of the operator is evaluated based on the collected data. However, such a collection method will bring obvious discomfort to the operator, and sweating will occur after wearing the electrodes for a long time, which in turn affects the accuracy of the collected data, resulting in the lack of reliability of the fatigue detection results based on these data. Therefore, how to improve the reliability of fatigue detection results has become a technical problem to be solved urgently.

[0066] Based on this, the embodiments of the present application provide a fatigue detection method and system for nuclear power plant operators, an electronic device, and a medium, aiming to improve the reliability of fatigue detection results.

[0067] The fatigue detection method and system for nuclear power plant operators, the electronic device, and the medium provided by the embodiments of the present application are specifically described through the following embodiments. First, the fatigue detection method for nuclear power plant operators in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results of theory, method, technology, and application system.

[0069] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The fatigue detection method provided by the embodiments of the present application relates to the field of nuclear power safety technology. The fatigue detection method provided by the embodiments of the present application can be applied to a terminal, or to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the fatigue detection method, etc., but is not limited to the above forms.

[0071] This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific embodiment of this application, when it comes to performing relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of this application need to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through methods such as pop-up windows or redirecting to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of this application will be obtained.

[0073] Figure 1 is an alternative flowchart of the fatigue detection method provided by the embodiments of this application, Figure 1 The method in [the figure] is applied to the fatigue detection device in the nuclear power control room and may include, but is not limited to, steps S101 to S106.

[0074] Step S101, obtain the original image data collected by the image acquisition device in the nuclear power control room for the operator.

[0075] Step S102, disassemble the original image data into video frames to obtain at least two consecutive original video frames.

[0076] Step S103, calculate the pixel color values for each original video frame to obtain the reference color values.

[0077] Step S104, construct a signal from all the reference color values according to the acquisition time sequence of the original video frames to obtain the reflected light change signal.

[0078] Step S105: Parse the reflected light change signal according to a preset signal frequency band to obtain the operator's heart rate data and respiratory rate data.

[0079] Step S106: Perform fatigue detection on the heart rate data and respiratory rate data according to a preset fatigue detection model.

[0080] Steps S101 to S106 shown in the embodiments of the present application. First, by using the image acquisition device deployed in the nuclear power control room to obtain the original image data, there is no need for the operator to wear special hardware devices and sampling electrodes, which avoids the discomfort brought to the operator and the problem of affecting data accuracy due to sweating during long-term electrode wearing, and improves the reliability from the data acquisition source. Then, disassemble the original image data into video frames and calculate and process the color values of the pixels to convert the image data into a reflected light change signal. Then, parse out the heart rate data and respiratory rate data according to the preset signal frequency band. This non-contact data acquisition method has minimal interference with the normal work of the operator and can continuously detect during the work process. Then, based on these accurately obtained physiological data, use the preset fatigue detection model to perform fatigue detection, which can more objectively and scientifically evaluate the fatigue degree of the operator. Compared with the traditional method, this solution does not rely on the subjective feelings of the operator, but is based on the actually collected physiological data for analysis and does not affect the body feeling of the operator, greatly improving the reliability of the fatigue detection result.

[0081] It should be noted that the fatigue detection method provided by the present application is implemented relying on the relevant hardware device system in the control room. Therefore, before explaining the steps, the fatigue detection system provided by the present application will be introduced first. Please refer to Figure 2 , Figure 2 is a schematic diagram of a fatigue detection system provided by the present application. The fatigue detection system includes a fatigue detection device, an image acquisition device, a pressure sensing device, and a body temperature measurement device. The pressure sensing device, the image acquisition device, and the body temperature measurement device are respectively connected to the fatigue detection device. It should be noted that the meaning of the connection between two devices includes that data communication can be achieved between the two devices, and it can also include a wired connection between the two devices. This embodiment does not strictly limit this.

[0082] In some embodiments, the image acquisition device collects image data of the operator in the nuclear power control room. The image acquisition device can be an acquisition device such as a camera or a depth camera that can capture RGB images. In the scenario of the nuclear power workstation in this embodiment, multiple control terminals will be arranged and deployed in the workstation control room for the operator to operate, and each control terminal is equipped with an interactive screen and an image acquisition device.

[0083] In some embodiments, the pressure sensing device is used to obtain the sitting posture pressure data of the operator. The pressure sensing device can be arranged in a flexible seat cushion. The pressure sensing device can be composed of a sensing module arranged in an array, a matrix switch array, a signal processing module, a Bluetooth communication module, and a power supply module. Exemplarily, the sensing module is a matrix of flexible thin-film pressure sensing units, the area of its sensing region is designed to be 40×40 square centimeters, and the sensing module is hidden in a latex pad of the same area, which matches well with the operator's seat. The matrix switch array is used to select a single pressure sensing unit to be sampled each time. The signal processing module is mainly composed of a signal conditioning and sampling circuit of the pressure sensing matrix, the designed sampling frequency is 10 Hz, and the output data is the time series of the sampling matrix of the pressure sensing units, which is stored in vector form. The Bluetooth communication module is responsible for transmitting the real-time sampling signal of the pressure sensing module to the fatigue detection device. The power supply module is responsible for supplying power to the other modules. All components can be hidden behind the seat cushion and the seat panel to create a data acquisition condition where the subject is unaware. According to the collected pressure matrix sequence data, the sitting posture pressure distribution of the human body is drawn, that is, the sitting posture pressure data. Four basic features related to fatigue are extracted from the pressure distribution obtained from the flexible pressure sensing seat cushion: the x and y coordinates of the left and right ischial tuberosities; by analyzing the time series characteristics of the basic features, the effective value and the energy of the first frequency node of the third level of the wavelet packet are calculated. Finally, a fatigue feature vector composed of multiple features is formed.

[0084] When the operator is sitting at the work station to work, the pressure sensing device can detect the pressure exerted by the operator on it, that is, the sitting posture pressure data, and then transmit the sitting posture pressure data to the fatigue detection device, so that it can judge the change of the operator's sitting posture (upright sitting, left tilt, right tilt, forward tilt, backward tilt, left forward tilt, right forward tilt, left backward tilt, right backward tilt, and vertical jitter) according to this data, so as to further determine the fatigue degree of the operator.

[0085] In some embodiments, the body temperature measuring device is used to measure the body temperature data of the operator, preferably a body temperature measuring device that measures in a non-contact manner, such as an infrared thermometer. The body temperature measuring device can be arranged in a small wearable device, such as a smart bracelet and a smart earphone, which can detect the body temperature data of the operator in real time, and at the same time, the operator will not feel uncomfortable during the work process. In some other embodiments, these small wearable devices can also be equipped with a positioning system, a blood oxygen saturation measuring device, and a heart rate measuring device, which can measure multiple data of the operator at the same time.

[0086] In some embodiments, the fatigue detection device is used to detect the fatigue of the operator according to the data transmitted by the above-mentioned image acquisition device, pressure sensing device, and body temperature measuring device. The fatigue detection device includes an electronic device for executing the fatigue detection method provided in the embodiments of the present application. The electronic device will be further elaborated in the following embodiments and will not be elaborated here.

[0087] In step S101 of some embodiments, the original image data is the video image of the operator collected by the image acquisition device. In some nuclear power plant control rooms, multiple control terminals will be deployed, which means that the operator will face different control panels during work. Since subsequent image processing needs to identify the facial features and other frontal physiological features of the operator, if there is only one acquisition angle for the image acquisition device, some of the original image data obtained may not be able to capture the front of the operator, resulting in the lack of relevant physiological data. To this end, multiple image acquisition devices can be deployed in the control room, and each image acquisition device corresponds to a control terminal (as Figure 2 shown). On this basis, for the steps of obtaining the original image data, please refer to Figure 3 , step S101 may include but is not limited to steps S301 to S305:

[0088] Step S301, determine the current acquisition device from multiple image acquisition devices.

[0089] Step S302, perform facial key point recognition on the initial image data collected by the current acquisition device to obtain the facial key points in the picture.

[0090] Step S303, determine the acquisition object information matching the initial image data according to the facial key points in the picture.

[0091] Step S304, in response to the acquisition object information indicating that the operator is not facing the current acquisition device directly, re-determine the current acquisition device from multiple image acquisition devices.

[0092] Step S305, in response to the acquisition object information indicating that the operator is facing the current acquisition device directly, determine the initial image data collected by the current acquisition device as the original image data.

[0093] In step S301 of some embodiments, an image that can capture the front of the operator among multiple image acquisition devices is used as the current acquisition device. At the beginning of the workstation startup, a camera can be selected as the initial current acquisition device according to the pre-set priority order or position order. This default selection can be determined based on the common working scenarios in the control room and the conventional operation postures of the operator. For example, the most important processing matters in the control room are basically displayed on the control terminal in the middle position. Then, when initializing, the image acquisition device in the middle position can be defaulted as the initial current acquisition device to improve the success rate and efficiency of the initial acquisition.

[0094] In step S302 of some embodiments, the initial image data is the image data collected by the current acquisition device for the operator. Facial key point recognition can be implemented based on facial feature point detection algorithms such as Dlib or MediaPipe. The facial key points in the image refer to the facial feature point data of the operator in the initial image data.

[0095] In step S303 of some embodiments, the collected object information may include whether the collected object corresponding to the initial image data is the operator himself / herself, and whether the operator faces the current acquisition device in the initial image data. Based on the recognized facial key points in the image, the facial features and the relative positional relationship of the facial features of the collected object can be determined. Through these geometric feature information, combined with the pre-established operator facial feature database or model, it is determined whether the collected object corresponding to the currently collected image data is the operator himself / herself. In the operator facial feature database, the facial feature templates or model parameters of each operator are stored, and these templates or parameters can be constructed based on features such as the distribution of facial key points and the shape of the facial contour. The facial geometric features calculated in the current image are compared and matched with the templates in the database, and a similarity score is calculated. If the similarity score exceeds the set threshold, it is considered that the collected object is the operator himself / herself, and at the same time, the related personal information, such as name, work number, etc., can be obtained as the collected object information.

[0096] At the same time, it is determined whether the collected object is facing the current acquisition device through the above-mentioned facial features. In this embodiment, when the forehead, the tip of the nose, and the left and right cheeks are simultaneously recognized in the initial image data, it can be determined that the collected object is currently facing the current acquisition device.

[0097] In step S304 of some embodiments, when the forehead, the tip of the nose, and the left and right cheeks are not recognized in the initial image data for a continuous predetermined period of time, it can be determined that the operator is not facing the current acquisition device, where the predetermined period of time can be 20 seconds.

[0098] When it is responded that the collected object information reflects that the operator is not facing the current acquisition device, the next image acquisition device can be sequentially selected as the current acquisition device in the order of the numbers of the image acquisition devices until the captured image can reflect that the operator is currently facing the corresponding image acquisition device, and this image acquisition device is used as the latest current acquisition device.

[0099] In some embodiments, if the facial key points of the operator are not recognized in the images collected by all the image acquisition devices, it can be determined that the operator is not at work at this time, and at this time, all the image acquisition devices can be controlled to enter the standby state to avoid frequent switching of the cameras.

[0100] In step S305 of some embodiments, for the convenience of database retrieval of video files, the original image data is saved in segments in hours. In some embodiments, to ensure the coherence of the original image data, as long as an image acquisition device is determined as the current acquisition device, the initially acquired image data is regarded as the original image data.

[0101] Steps S301 to S305 illustrated in the embodiments of the present application can comprehensively cover the operator's working area by reasonably distributing and installing multiple image acquisition devices, ensuring that there are suitable devices for image acquisition in all scenarios, and improving the comprehensiveness and reliability of data acquisition. Facial key point recognition of the original image data can accurately obtain the facial feature point data of the operator, accurately identify whether the acquisition object is the operator himself, and at the same time determine whether the operator is facing the acquisition device directly through specific facial features, realizing accurate recognition of the identity and posture of the acquisition object. When it is found that the operator is not facing the current acquisition device, the image acquisition devices are sequentially switched in the order of numbers, ensuring that the finally obtained original image data includes a relatively complete face image of the operator, thereby improving the reliability of the original image data.

[0102] In step S102 of some embodiments, using professional video processing software or algorithm libraries, such as OpenCV, etc., perform video frame disassembling operations on the obtained original image data, and the finally obtained video frames are the original video frames. During the disassembling process, ensure accurate extraction of consecutive original video frames according to the frame rate of the camera, and avoid frame loss or frame error phenomena. For each frame of image, key data such as pixel information, image format, and time stamp should be completely retained for subsequent accurate analysis and processing.

[0103] In step S103 of some embodiments, please refer to Figure 4 , in some embodiments, step S103 may include but is not limited to steps S401 to S403:

[0104] Step S401, perform region extraction on the original video frame to obtain the target region.

[0105] Step S402, extract the color value of each pixel in the target region to obtain the pixel color value.

[0106] Step S403, perform average value calculation based on all the pixel color values to obtain the reference color value.

[0107] In step S401 of some embodiments, the target region includes a first target sub-region and a second target sub-region. The first target sub-region is a local region of the operator's face, and the second target sub-region is a local region of the operator's chest. In this embodiment, the second target sub-region is the region corresponding to the operator's heart. Specifically, the first target sub-region may include the forehead, local regions of the left cheek, and local regions of the right cheek. It can be understood that the method of selecting multiple sub-regions as the target region together has the following advantages in subsequent image processing: 1. When the face is partially occluded (such as when the hair covers the face when the head is lowered, and when the palm covers the face when the operator supports the face), selecting multiple first target sub-regions can make full use of the unoccluded regions, thus avoiding the loss of data to be processed. 2) Large tilts and shakes of the face may cause some target sub-regions to be outside the image frame, resulting in corresponding data loss. 3) Affected by factors such as the light source angle and light changes, the target regions of different positions and sizes on the face are affected differently, and the signal quality is also different. However, it can be considered that these target regions contain heartbeat information that is homologous but of different intensities, which lays a foundation for subsequent obtaining the correct heart rate information through signal decomposition.

[0108] For the method of target region extraction, relevant operators can directly select the region of interest through the input device, or it can be determined by a pre-trained target recognition model. This method can adjust the range of the target region in real time according to the position changes of the facial key points, ensuring that the effective region related to the physiological signal can always be accurately extracted.

[0109] In step S402 of some embodiments, the pixel color value is the RGB value of the current pixel.

[0110] In step S403 of some embodiments, the reference color value is the average value of all pixels in the RGB color space in the target region. The reference color value satisfies the following analytical formula:

[0111]

[0112] Among them, m represents the sequence of the original video frames. k represents the sequence of the target sub-regions. S k (m) represents the reference color value. ROI k (x,y,m) represents the pixel color value at the pixel coordinates (x,y) in the k-th target sub-region. R(m), G(m), B(m) represent the color values corresponding to the red, green, and blue channels in the m-th original video frame. PN k represents the total number of pixels in the k-th target sub-region.

[0113] It should be noted that when the target area is composed of multiple sub-areas, for each sub-area, the corresponding reference color value is calculated separately, that is, the average pixel color value of the sub-area.

[0114] Steps S401 to S403 shown in the embodiments of the present application can focus on the key areas closely related to the operator's physiological signals by extracting the target area from the original video frame, which helps to capture more accurately the color features related to physiological changes subsequently and lays a foundation for obtaining effective physiological signal data. The color value of each pixel in the target area is extracted, and the reference color value is calculated according to the color values of all pixels, effectively integrating the color information in the area, reducing the influence of individual pixel noise and local color changes, and obtaining a characteristic value that can comprehensively reflect the overall change trend of the color of the target area. This reference color value can be used as a stable and reliable indicator in subsequent signal analysis, improving the accuracy and stability of the analysis of heart rate and respiratory rate data.

[0115] In step S104 of some embodiments, when the target area is composed of a first target sub-area and a second target sub-area, a timing signal is generated for the reference color value corresponding to the first target sub-area according to the acquisition timing of the original video frame to obtain a facial reflected light change signal. At the same time, a timing signal is generated for the reference color value corresponding to the second target sub-area according to the acquisition timing to obtain a chest reflected light change signal. It should be noted that the facial reflected light change signal and the chest reflected light change signal can be remote photoplethysmography (rPPG) signals. The principle of rPPG signals is based on the camera capturing the slight color changes caused by blood flowing through the facial skin. Since blood flow is affected by heart activity, analyzing the fluctuation frequency of this color change over time can reflect heart rate information. It should be noted that an rPPG signal is constructed for each target sub-area. Then, the reflected light change signal corresponding to the target area is obtained according to the facial reflected light change signal and the chest reflected light change signal.

[0116] Please refer to Figure 5 , in step S105 of some embodiments, the preset signal frequency band includes a first signal sub-band and a second signal sub-band. Specifically, the first signal sub-band can be [0.7Hz, 4Hz], and the second signal sub-band can be [0.1Hz, 0.4Hz]. Step S105 may include but is not limited to steps S501 to S504:

[0117] Step S501, extracting a signal from the facial reflected light change signal according to the first signal sub-band to obtain a candidate reflected light change signal.

[0118] Step S502: Decompose the candidate reflected light change signal according to the preset decomposition constraint conditions to obtain at least two candidate reflected photon signals.

[0119] Step S503: Perform frequency-domain analysis based on the candidate reflected photon signals to obtain heart rate data.

[0120] Step S504: Perform frequency-domain analysis on the chest reflected light change signal according to the second signal sub-band to obtain respiration rate data.

[0121] In step S501 of some embodiments, the candidate reflected light change signal is the facial reflected light change signal within the first signal sub-band in terms of frequency.

[0122] In step S502 of some embodiments, in order to fully extract the information more closely related to the heartbeat in the signal, the variational mode extraction (VME) algorithm can be used to decompose the candidate reflected light change signal. However, if the candidate reflected light change signal is over-decomposed, not only will the computational amount increase, but also excessive noise interference may be introduced. If the signal decomposition is insufficient, the information related to the heartbeat cannot be fully extracted, thereby reducing the accuracy of subsequent signal analysis. Therefore, it is necessary to determine the termination condition of signal decomposition, that is, the decomposition constraint condition. The signal after the candidate reflected light change signal is decomposed is the candidate reflected photon signal.

[0123] Specifically, it is necessary to determine the obviousness of the main peak of the candidate reflected light change signal within [0.7, 4 Hz], which can be measured according to the ratio between the amplitude of the main peak of the candidate reflected light change signal and the average amplitude of the first signal sub-band. When the obviousness of the main peak is greater than or equal to the preset threshold, it indicates that the quality of the candidate reflected light change signal is good and does not require excessive decomposition. In this case, the termination condition of decomposition is where Er represents the energy of the residual signal after decomposition, and Es represents the total energy of the candidate reflected light change signal. If the obviousness of the main peak is less than the preset threshold, it indicates that the quality of the candidate reflected light change signal is poor and there is more noise, and sufficient decomposition is required. The corresponding decomposition termination condition is

[0124] In step S503 of some embodiments, please refer to Figure 6 , step S503 includes but is not limited to steps S601 to S604:

[0125] Step S601: Perform dimensionality reduction transformation processing on each candidate reflected photon signal to obtain a reconstructed signal.

[0126] Step S602: Perform time-domain characteristic analysis on the reconstructed signal to obtain a pulse time-domain signal.

[0127] Step S603: Perform a frequency-domain transformation on the pulse time-domain signal to obtain a pulse frequency-domain signal.

[0128] Step S604: Determine the heart rate data based on the pulse frequency-domain signal.

[0129] In step S601 of some embodiments, multi-view canonical correlation analysis (MCCA) can be used to perform dimensionality reduction and feature extraction on the candidate anti-photon signal, and output multiple signals containing real pulse information, that is, reconstructed signals.

[0130] In step S602 of some embodiments, perform a time-domain characteristic analysis on the reconstructed signal. The time-domain characteristics can be the peak-to-peak time interval of the signal, the change rate of the signal, and the average amplitude of the signal, etc. These characteristics can evaluate the quality of the reconstructed signal. Exemplarily, perform a timing analysis on each reconstructed signal, and select the reconstructed signal with the lowest signal change rate as the pulse time-domain signal.

[0131] In step S603 of some embodiments, a fast Fourier transform can be performed on the pulse time-domain signal to obtain a pulse frequency-domain signal.

[0132] In step S604 of some embodiments, the maximum peak value of the pulse frequency-domain signal in the frequency domain is the frequency corresponding to the heartbeat, and the heart rate data of the operator can be calculated according to the following analytical formula:

[0133] HR = f hr × 60 (2),

[0134] where HR represents the heart rate data, and f hr represents the maximum peak value of the pulse frequency-domain signal.

[0135] In some other embodiments, the time interval between heartbeats (R-R interval) can be obtained based on the peak value of the pulse frequency-domain signal. Subsequently, when performing fatigue detection in the model, heart rate variability (HRV) feature analysis can be performed based on this interval to obtain multiple feature information, as summarized in Table 1, thereby further improving the reliability of the fatigue detection result.

[0136]

[0137] Table 1

[0138] Steps S601 to S604 illustrated in the embodiments of the present application, through dimensionality reduction transformation processing on candidate reflected photon signals, retain key information, reduce the amount of data processing and computational complexity, and improve the efficiency of subsequent analysis. This helps to remove redundant information and noise interference, making the reconstructed signal better highlight the key components reflecting pulse characteristics, laying a good foundation for accurate analysis of pulse signals. Analyzing the time-domain characteristics of the reconstructed signal can deeply explore the variation law of the pulse signal in the time dimension. Converting the pulse time-domain signal into a frequency-domain signal and using the fast Fourier transform reveals the spectral distribution of the signal in the frequency domain, and finally accurately finds the frequency peak corresponding to the heart rate, providing a direct basis for determining heart rate data.

[0139] In step S504 of some embodiments, the part with a frequency band of the second signal sub-band (i.e., [0.1 Hz, 0.4 Hz]) is extracted from the reflected light change signal of the chest, and the fast Fourier transform is performed on this part of the signal. The frequency corresponding to the maximum peak in the obtained frequency distribution is the breathing frequency, and the breathing rate can be calculated according to the following analytical formula:

[0140] RR = f rr × 60 (3),

[0141] where RR represents the number of breaths per minute, that is, the breathing rate. f rr represents the breathing frequency.

[0142] Steps S501 to S504 illustrated in the embodiments of the present application, through precise extraction of the facial reflected light change signal according to the first signal sub-band, can effectively focus on the signal components related to the heart rate and exclude the interference of other irrelevant frequency signals. Reasonably decomposing the candidate reflected light change signal according to the preset decomposition constraint conditions can decompose the complex reflected light change signal into multiple more characteristic candidate reflected photon signals, which helps to deeply explore the heart rate information in the signal. Based on the candidate reflected photon signals for frequency-domain analysis, the heart rate data is accurately determined using the frequency-domain characteristics. At the same time, the breathing rate data is obtained by performing frequency-domain analysis on the chest reflected light change signal according to the second signal sub-band, realizing non-contact detection of this important physiological index of breathing. Combined with facial heart rate detection, a multi-modal physiological signal detection system is formed.

[0143] In some other embodiments, other modal data of the original image data will also be detected, such as the eye closure state, head posture, and predicted gaze area of the operator.

[0144] Exemplarily, the closed-eye state of the operator can be achieved in the following manner. Taking the upper-left corner point of the sample image as the origin, with the horizontal direction as the x-axis and the vertical direction as the y-axis, a training set can be pre-constructed by collecting multiple sample pictures and obtaining facial key points for the sample pictures. Determine the key points of the left corner of the left eye and the right corner of the right eye on the portrait in the sample image, and calculate the coordinates a(ax, ay) of the rotation center point based on the coordinates of these two key points, as shown in the following analytical formula:

[0145]

[0146] Among them, P37.x and P37.y respectively represent the abscissa and ordinate of the key point of the left corner of the eye. P46.x and P46.y respectively represent the abscissa and ordinate of the key point of the right corner of the eye. Align all the sample images according to the coordinates of the above rotation center point and label them. In this embodiment, the labels are open eyes and closed eyes. Then, an eye-closure detection model based on a neural network can be constructed and trained using the pre-constructed open / closed-eye detection data set with labels. Finally, the original video frames are detected according to the trained eye-closure detection model to determine the closed-eye state of the operator. Exemplarily, the specifications of the original video frames are processed to a size of 3×32×32 and used as the input to the neural network. 6 convolutional kernels are used to perform convolutional operations on the input sample images. The size of the convolutional kernel is 5×5, the stride is 1, and no edge padding is performed, resulting in a feature vector with a specification of 6×28×28. Perform a max-pooling operation on it. The size of the convolutional kernel is 2×2, the stride is 2, and no edge padding is performed, and the specification of the feature vector is converted to 6×14×14. Use 16 convolutional kernels to perform convolutional operations. The size of the convolutional kernel is 5×5, the stride is 1, and no edge padding is performed, and the specification of the feature vector is converted to 16×10×10. Perform a max-pooling operation on it. The size of the convolutional kernel is 2×2, the stride is 2, and no edge padding is performed, and the specification of the feature vector is converted to 16×5×5. Use 120 convolutional kernels to perform convolutional operations. The size of the convolutional kernel is 5×5, the stride is 1, and no edge padding is performed, and the specification of the feature vector is converted to 120×1×1. Input the obtained feature vector into the first fully connected layer for full connection calculation. The number of neurons in the first fully connected layer is 120, resulting in a feature vector with a specification of 1×120. Input it into the second fully connected layer for full connection calculation. The number of neurons in the second fully connected layer is 2, corresponding to the labels of two categories, resulting in a feature vector with a specification of 1×2. Use the softmax function to perform a normalization operation on it to obtain the probability that the input eye picture is open or closed.

[0147] Exemplarily, the acquisition of the head pose can be achieved in the following manner. In the above embodiment, the facial key points and their coordinates in the picture have been obtained. Based on these key point coordinates, combined with the general 3D coordinate model of head key points, the rotation matrix of the head is calculated through the N-point perspective pose solution algorithm. The rotation matrix describes the rotation state of the head relative to the camera. Then, this matrix is used to represent the head pose as the pitch angle, yaw angle, and roll angle in the spatial coordinate system, thereby completely characterizing the three-dimensional pose of the head. In the above calculation process, the calibration of the image acquisition device is a key step. The Zhang Zhengyou calibration method can be used to calibrate the internal parameter matrix and distortion parameters of the image acquisition device, and complete the coordinate transformation relationship between the world coordinate system and the camera coordinate system. According to the calibration result, the facial key points are aligned with the standard face model, and the head rotation matrix and head translation vector are calculated. Finally, the head pose can be clearly described.

[0148] Exemplarily, the acquisition of the predicted fixation area can be achieved in the following manner. The publicly available dataset DDGC-DB1 is used as the training set for the training model. The specifications of each sample in the training set are uniformly adjusted to 224×224. The facial image is intercepted from the scaled image. Denote the width of the facial image as L1 and the height as H1. With the geometric center of the eye region as the center, the binocular image is intercepted. Denote the distance between the two eye corners as L2 and the height as H2. The intercepted facial image and binocular image are respectively used for feature extraction using the convolutional layer of the VGG16 neural network to obtain three 1×4096 feature vectors, denoted as ξ, ψ, and γ respectively, and the weighted sum vector Γ is calculated. Calculate the Euclidean distance o1 and Euclidean distance o2 between the vectors ξ and ψ, as shown in the following analytical formula.

[0149]

[0150]

[0151] Γ = ξ + o1ψ + o2γ (8),

[0152] where k = 1, 2, 3, …, 4095. ξk represents the k-th element value of the vector ξ, ψk represents the k-th element value of the vector ψ, and γk represents the k-th element value of the vector γ.

[0153] After the vector Γ is calculated through two fully connected layers, the result is normalized using the softmax function to obtain the output vector R. The element values in the vector R are the predicted probability values of various categories, and the element position where the maximum value is located is the predicted result of the fixation area.

[0154] Before step S106 of some embodiments, sitting pressure data collected by a pressure sensing device in a nuclear power control room is obtained. The sitting pressure data is the pressure exerted by an operator on the pressure sensing device, which has been explained in detail in the embodiments of the fatigue detection system and will not be elaborated here.

[0155] In some embodiments, body temperature data collected by a body temperature measuring device in the nuclear power control room for the operator can also be obtained simultaneously.

[0156] In step S106 of some embodiments, fatigue detection is performed on the heart rate data and respiratory rate data according to a preset fatigue detection model, and the corresponding fatigue state of the operator can be obtained. The fatigue state can be fatigue and wakefulness.

[0157] In addition to performing fatigue detection on the respiratory rate and heart rate data, the fatigue detection model can also use any combination of the above sitting pressure data, body temperature data, head posture, eye closure state, predicted fixation area, and HRV features obtained from the heart rate data as the input of the fatigue detection model. Corresponding weight parameters are preset for different modality data, and the fatigue degree of the operator is comprehensively evaluated by combining the prediction results of the model and the weight parameters.

[0158] The embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned fatigue detection method is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0159] Please refer to Figure 7 , Figure 7 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0160] A processor 701, which can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0161] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 702 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 702, and the processor 701 is used to call and execute the fatigue detection method of the embodiments of this application;

[0162] The input / output interface 703 is used to implement information input and output;

[0163] The communication interface 704 is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0164] The bus 705 transmits information between various components of the device (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0165] Among them, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are communicatively connected to each other inside the device through the bus 705.

[0166] The embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned fatigue detection method is implemented.

[0167] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely provided relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The fatigue detection method and system, electronic device and medium for nuclear power plant operators provided by the embodiments of the present application first obtain the original image data by using the image acquisition device deployed in the nuclear power control room. There is no need for operators to wear special hardware devices and sampling electrodes, which avoids the discomfort caused to operators and the problem of affecting data accuracy due to sweating after wearing electrodes for a long time, and improves the reliability from the source of data acquisition. Then, the original image data is disassembled into video frames, and the color values of the pixels are calculated and processed to convert the image data into a reflected light change signal. Then, the heart rate data and respiratory rate data are parsed according to the preset signal frequency band. This non-contact data acquisition method has minimal interference with the normal work of operators and can continuously detect during the work process. Then, based on these accurately acquired physiological data, the preset fatigue detection model is used for fatigue detection, which can more objectively and scientifically evaluate the fatigue degree of operators. Compared with the traditional method, the present solution does not rely on the subjective feelings of operators, but analyzes based on the actually collected physiological data and does not affect the physical feeling of operators, greatly improving the reliability of the fatigue detection results.

[0169] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0170] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or combine some steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware and their appropriate combinations.

[0173] In the description of the present application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0174] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c may mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c may be single or multiple.

[0175] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned unit division is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store programs.

[0179] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for detecting fatigue of a nuclear power plant operator, characterized in that: A fatigue detection device applied to a nuclear power control room, the method comprising: Acquiring original image data collected by the image acquisition device of the nuclear power control room from the operator; Decomposing the original image data into video frames to obtain at least two continuous original video frames; Calculating pixel color values ​​for each of the original video frames to obtain reference color values; Constructing signals of all the reference color values ​​according to the acquisition timing of the original video frame to obtain a reflected light change signal; Performing signal analysis on the reflected light change signal according to a preset signal frequency band to obtain the heart rate data and breathing rate data of the operator; Fatigue detection is performed on the heart rate data and the respiratory rate data according to a preset fatigue detection model.

2. The fatigue detection method according to claim 1, characterized in that: The step of calculating the pixel color value of each original video frame to obtain a reference color value includes: Performing region extraction on the original video frame to obtain a target region; Extracting the color value of each pixel in the target area to obtain the pixel color value; An average value is calculated based on all the pixel color values ​​to obtain the reference color value.

3. The fatigue detection method according to claim 2, characterized in that: The target area includes a first target sub-area and a second target sub-area, the first target sub-area is a local area of ​​the operator's face, and the second target sub-area is a local area of ​​the operator's chest; the signal construction of all the reference color values ​​according to the acquisition timing of the original video frame to obtain the reflected light change signal includes: Generating a timing signal for the reference color value corresponding to the first target sub-area according to the acquisition timing to obtain a facial reflected light change signal; Generating a time sequence signal for the reference color value corresponding to the second target sub-area according to the acquisition time sequence to obtain a chest reflected light change signal; The reflected light change signal is obtained according to the facial reflected light change signal and the chest reflected light change signal.

4. The fatigue detection method according to claim 3, characterized in that: The preset signal frequency band includes a first signal sub-frequency band and a second signal sub-frequency band; the signal analysis of the reflected light change signal according to the preset signal frequency band to obtain the heart rate data and breathing rate data of the operator includes: performing signal extraction on the facial reflected light change signal according to the first signal sub-band to obtain a candidate reflected light change signal; Decomposing the candidate reflected light change signal according to a preset decomposition constraint condition to obtain at least two candidate reflected photon signals; Perform frequency domain analysis on the candidate reflected photon signal to obtain the heart rate data; The respiratory rate data is obtained by performing frequency domain analysis on the chest reflected light change signal according to the second signal sub-band.

5. The fatigue detection method according to claim 4, characterized in that: The performing frequency domain analysis according to the candidate reflected photon signal to obtain the heart rate data includes: Performing dimensionality reduction transformation processing on each of the candidate reflected photon signals to obtain a reconstructed signal; Performing time domain characteristic analysis on the reconstructed signal to obtain a pulse time domain signal; Performing frequency domain transformation on the pulse time domain signal to obtain a pulse frequency domain signal; The heart rate data is determined according to the pulse frequency domain signal.

6. The fatigue detection method according to any one of claims 1 to 5, characterized in that: A plurality of the image acquisition devices are arranged in the nuclear power control room; The method of obtaining the original image data collected by the image acquisition device of the nuclear power control room from the operator comprises: Determining a current acquisition device from a plurality of image acquisition devices; Perform facial key point recognition on the initial image data acquired by the current acquisition device to obtain facial key points of the picture; Determining acquisition object information matching the initial image data according to the facial key points of the picture; In response to the acquisition object information reflecting that the operator is not facing the current acquisition device, re-determining the current acquisition device from a plurality of image acquisition devices; In response to the acquisition object information reflecting that the operator is facing the current acquisition device, initial image data acquired by the current acquisition device is determined as the original image data.

7. The fatigue detection method according to any one of claims 1 to 5, characterized in that: Before performing fatigue detection on the heart rate data and the respiratory rate data according to the preset fatigue detection model, the method further includes: Acquire sitting posture pressure data collected by the pressure sensing device in the nuclear power control room; wherein the sitting posture pressure data is the pressure applied by the operator to the pressure sensing device; Acquiring body temperature data of the operator collected by a body temperature measuring device in the nuclear power control room; The step of performing fatigue detection on the heart rate data and the respiratory rate data according to a preset fatigue detection model further includes: Fatigue detection is performed on the heart rate data, the breathing rate data, the sitting posture pressure data and the body temperature data according to the fatigue detection model.

8. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the fatigue detection method according to any one of claims 1 to 7 when executing the computer program.

9. A fatigue detection system, characterized in that: The fatigue detection system comprises: Fatigue detection device; wherein the fatigue detection device comprises the electronic device as claimed in claim 8; An image acquisition device is used to acquire image data from operators in a nuclear power control room; A pressure sensor device, used for obtaining the sitting pressure data of the operator; A body temperature measuring device, used to measure the body temperature data of the operator; Wherein, the pressure sensing device, the image acquisition device and the body temperature measuring device are respectively connected to the fatigue detection device.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the fatigue detection method according to any one of claims 1 to 7 is implemented.

Images