Target object state monitoring and early warning method and device, computer equipment, readable storage medium and program product
By acquiring psychological state questionnaires and collecting eye-tracking data and facial expression images in real time, and using weighted algorithms to assess the status of railway station operators, the problem of low efficiency in traditional manual monitoring has been solved, enabling rapid and automated early warning and intervention, and ensuring the safe operation of railway stations.
Patent Information
- Application Number
- CN202511458907.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-13
- Publication Date
- 2026-01-06
AI Technical Summary
Traditional methods for monitoring the status of railway station operators rely on manual judgment, which is slow, labor-intensive, and inefficient, making it difficult to ensure safe operation.
By acquiring feedback from psychological state questionnaires, eye-tracking data, and facial expression images, a weighted algorithm is used to comprehensively assess the status of operations personnel and send out early warning signals in a timely manner.
It enables rapid and automated monitoring of the status of operational personnel, reduces human-caused accidents, and improves the safety and operational efficiency of railway stations.
Smart Images
Figure CN121278552A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of safety monitoring technology, and in particular to a method, device, computer equipment, computer-readable storage medium, and computer program product for monitoring and early warning of the status of a target object. Background Technology
[0002] As high-traffic areas, railway stations require their staff to maintain a high level of focus at all times. Prolonged work can cause both physical and mental strain on these staff. Therefore, a method for monitoring the physical condition of railway station staff is urgently needed to ensure their well-being.
[0003] Traditional methods for monitoring the condition of operational personnel require dedicated monitoring staff to patrol and observe the stations, assessing the physical condition of staff through visual inspection. However, this method relies heavily on the subjective judgment of the monitoring personnel, resulting in slow response times, high manpower requirements, and a lack of systematic approach. Consequently, it suffers from low monitoring efficiency and delayed early warnings, making it difficult to guarantee the safe operation of railway stations. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for monitoring and early warning of the status of a target object, in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides a method for monitoring and early warning of the state of a target object, including:
[0006] Obtain the psychological status questionnaire responses from the target subject before they start their job, and analyze the psychological status of the target subject based on the psychological status questionnaire responses.
[0007] If the psychological state is abnormal, an early warning message is generated based on the type of abnormality. If the psychological state is not abnormal, the target object to be tested is identified as an on-duty target object.
[0008] The eye-tracking data and facial expression images of the on-duty target object are collected through a terminal device during the operation. An eye trajectory map and an eye-tracking heat map are generated based on the eye-tracking data. A facial expression shape model is constructed based on the facial expression images. Facial expression feature vectors are extracted through the facial expression shape model.
[0009] The on-duty status of the target object is comprehensively evaluated by a weighted algorithm based on the eye trajectory map, the eye movement heat map, and the facial expression feature vector. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0010] In one embodiment, constructing a facial expression shape model based on the facial expression image includes:
[0011] In the facial expression image, the position of facial features is estimated with the top of the head as a reference point, and marker points are evenly set on the contour of each facial feature. The central axis of the facial expression image is fitted according to the midpoint of the line connecting the two pupils and the center of the mouth. The facial expression image is divided into two symmetrical parts based on the central axis. The marker points symmetrical with respect to the central axis are adjusted to the same horizontal line, and the facial expression shape model is constructed based on the adjusted marker points.
[0012] In one embodiment, the step of extracting facial expression feature vectors through the facial expression shape model includes:
[0013] In the facial expression shape model, the facial expression image is divided into multiple different feature candidate regions according to the left eye and left eyebrow, the right eye and right eyebrow, and the mouth. For each feature candidate region, the difference image method is used to perform a difference operation on all image sequences in the facial expression image and neutral expression images in the database to obtain the mean difference value corresponding to each feature candidate region. The facial expression feature vector is extracted from the image sequence with the largest mean difference value in each feature candidate region.
[0014] In one embodiment, generating an eye trajectory map and an eye movement heatmap based on the eye movement data includes:
[0015] The eye movement data is classified and feature-extracted according to eye movement trajectory, fixation time, and fixation frequency to obtain eye movement trajectory feature vector, fixation time feature vector, and fixation frequency feature vector; eye movement tracking analysis is performed on the eye movement trajectory feature vector, and the eye trajectory map is generated based on the analysis results; the eye movement heatmap is drawn based on the fixation time feature vector and the fixation frequency feature vector.
[0016] In one embodiment, the step of analyzing the psychological state of the target subject based on the psychological state questionnaire includes:
[0017] The psychological state questionnaire is used to analyze the personality, emotional state, cognitive state, and psychological stress of the test subject; based on the personality, emotional, cognitive, and psychological stress states, the analysis results of the psychological state are obtained. In one embodiment, the method further includes:
[0018] Based on preset threshold conditions, it is determined whether the personality condition, emotional condition, cognitive condition, and psychological stress condition are abnormal; if at least one of the personality condition, emotional condition, cognitive condition, and psychological stress condition is abnormal, it is determined that the psychological state of the target subject to be tested has the abnormal condition.
[0019] Secondly, this application also provides a target object status monitoring and early warning device, including:
[0020] The status analysis module is used to obtain the psychological status questionnaire responses of the target subject before they start their job, and to analyze the psychological status of the target subject based on the psychological status questionnaire responses.
[0021] The first early warning module is used to generate early warning information based on the abnormality type corresponding to the abnormality if the psychological state is abnormal; and to determine the target object to be tested as the target object on duty if the psychological state is not abnormal.
[0022] The feature extraction module is used to acquire eye movement data and facial expression images collected by the on-duty target object through a terminal device during the operation, generate an eye trajectory map and an eye movement heat map based on the eye movement data, construct a facial expression shape model based on the facial expression images, and extract facial expression feature vectors through the facial expression shape model.
[0023] The second early warning module is used to comprehensively evaluate the on-duty status of the on-duty target object based on the eye trajectory map, the eye movement heat map, and the facial expression feature vector through a weighted algorithm. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0024] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0025] The system obtains the psychological state questionnaire of the target subject before they start work, and analyzes the psychological state of the target subject based on the questionnaire. If there are abnormalities in the psychological state, an early warning message is generated according to the type of abnormality. If there are no abnormalities in the psychological state, the target subject is identified as an on-duty target subject. The system obtains eye movement data and facial expression images of the on-duty target subject through a terminal device during the work process. An eye trajectory map and an eye movement heatmap are generated based on the eye movement data. A facial expression shape model is constructed based on the facial expression images, and facial expression feature vectors are extracted from the facial expression shape model. The on-duty status of the on-duty target subject is comprehensively evaluated through a weighted algorithm based on the eye trajectory map, the eye movement heatmap, and the facial expression feature vectors. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0026] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0027] The system obtains the psychological state questionnaire of the target subject before they start work, and analyzes the psychological state of the target subject based on the questionnaire. If there are abnormalities in the psychological state, an early warning message is generated according to the type of abnormality. If there are no abnormalities in the psychological state, the target subject is identified as an on-duty target subject. The system obtains eye movement data and facial expression images of the on-duty target subject through a terminal device during the work process. An eye trajectory map and an eye movement heatmap are generated based on the eye movement data. A facial expression shape model is constructed based on the facial expression images, and facial expression feature vectors are extracted from the facial expression shape model. The on-duty status of the on-duty target subject is comprehensively evaluated through a weighted algorithm based on the eye trajectory map, the eye movement heatmap, and the facial expression feature vectors. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0028] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0029] The system obtains the psychological state questionnaire of the target subject before they start work, and analyzes the psychological state of the target subject based on the questionnaire. If there are abnormalities in the psychological state, an early warning message is generated according to the type of abnormality. If there are no abnormalities in the psychological state, the target subject is identified as an on-duty target subject. The system obtains eye movement data and facial expression images of the on-duty target subject through a terminal device during the work process. An eye trajectory map and an eye movement heatmap are generated based on the eye movement data. A facial expression shape model is constructed based on the facial expression images, and facial expression feature vectors are extracted from the facial expression shape model. The on-duty status of the on-duty target subject is comprehensively evaluated through a weighted algorithm based on the eye trajectory map, the eye movement heatmap, and the facial expression feature vectors. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0030] The aforementioned methods, devices, computer equipment, computer-readable storage media, and computer program products for monitoring and early warning of the status of target objects, targeting the easily fluctuating psychological state in the short term, construct a rapid pre-job status detection method. Based on the psychological state questionnaire responses of the target objects before starting work, the method analyzes the psychological state of the target objects and provides early warnings for those with abnormal psychological states. Secondly, during operation, eye-tracking data and facial expression images of the on-duty target objects are collected through terminal equipment. Information fusion technology and multi-model result coupling methods are used, combined with deep learning algorithms, to comprehensively evaluate the on-duty status of the target objects, achieving dynamic monitoring of the on-duty personnel's status and workload. This application provides a systematic status monitoring and early warning method for target objects, enabling timely detection and intervention of adverse states of on-duty personnel to reduce human-caused accidents, thereby improving monitoring efficiency and avoiding the problem of delayed early warning, which is conducive to ensuring the safe operation of railway stations. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is an application environment diagram of a target object status monitoring and early warning method in one embodiment;
[0033] Figure 2 This is a flowchart illustrating a method for monitoring and issuing early warnings of the state of a target object in one embodiment;
[0034] Figure 3 This is a flowchart illustrating the steps involved in constructing a facial expression shape model in one embodiment.
[0035] Figure 4 This is a flowchart illustrating a method for monitoring and issuing early warnings of the status of a target object in a specific embodiment.
[0036] Figure 5 This is a structural block diagram of a target object status monitoring and early warning device in one embodiment;
[0037] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0039] The target object status monitoring and early warning method provided in this application embodiment can be applied to, for example, Figure 1 The application environment shown illustrates this. In this environment, the terminal can communicate with the server via a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated onto the server or located on the cloud or other network servers. In situations such as... Figure 1 In the application environment shown, the terminal can be, but is not limited to, a wearable device or peripheral device configured on the target object under test. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0040] In one embodiment, such as Figure 2 As shown, a method for monitoring and early warning of the state of a target object is provided, which can be applied to... Figure 1 The method may include the following steps:
[0041] Step S201: Obtain the psychological status questionnaire of the target subject before starting work, and analyze the psychological status of the target subject based on the psychological status questionnaire.
[0042] The target group to be tested can be station operators in various positions.
[0043] Among them, psychological state questionnaires may include the Eysenck Personality Questionnaire, the Calter Personality Questionnaire, the Depression Self-Rating Questionnaire, the State-Trait Anxiety Questionnaire, the Wisconsin Card Sorting Questionnaire, the Mental Cognitive Ability Questionnaire, the Coping Style Questionnaire, and the Social Support Rating Questionnaire, etc.
[0044] Specifically, in response to the received status monitoring instruction for the target object, the server obtains the target object's psychological status questionnaire based on the psychological status questionnaire feedback before starting work, and analyzes the target object's personality, emotional state, cognitive state, and psychological stress state based on the psychological status questionnaire.
[0045] In step S202, if there is an abnormality in the psychological state, an early warning message is generated according to the abnormality type corresponding to the abnormality; if there is no abnormality in the psychological state, the target to be tested is identified as the target on duty.
[0046] Specifically, if the server identifies at least one of the following as abnormalities in the target object's personality, emotional state, cognitive state, and psychological stress state, it determines that the target object's psychological state is abnormal and generates an early warning message based on the type of abnormality. If the server does not identify any abnormalities in the target object's psychological state, it determines the target object as an on-duty target object.
[0047] Step S203: Obtain eye movement data and facial expression images collected by the on-duty target object through the terminal device during the operation. Generate eye trajectory map and eye movement heat map based on the eye movement data. Construct a facial expression shape model based on the facial expression images. Extract facial expression feature vectors through the facial expression shape model.
[0048] Among them, the terminal device can be wearable devices such as smartwatches and smart glasses, as well as peripheral devices such as cameras and video capture devices.
[0049] Specifically, in response to the received status monitoring command for the on-duty target, the server acquires eye movement data and facial expression images collected by the on-duty target through the terminal device during the operation. The eye movement data is plotted into an eye trajectory map according to the eye movement trajectory. Then, an eye movement heat map is generated based on the fixation time and fixation frequency in the eye movement data. Finally, a facial expression shape model is constructed based on the facial expression image, and facial expression feature vectors are extracted through the facial expression shape model.
[0050] Step S204: Based on the eye trajectory map, eye movement heat map, and facial expression feature vector, the on-duty status of the target object is comprehensively evaluated using a weighted algorithm. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0051] The core idea of the weighted algorithm is to influence the contribution of each data point to the overall result based on its weight. The weight is typically a numerical value used to express the importance of that data point in a particular calculation.
[0052] Warning signals include, but are not limited to, sound signals and visual symbols.
[0053] Specifically, the server uses a weighted algorithm to comprehensively evaluate the on-duty status of the target object based on the weights corresponding to the eye trajectory map, eye movement heat map, and facial expression feature vector, and obtains an on-duty status evaluation value. If the on-duty status evaluation value meets the threshold condition, the on-duty status of the target object is determined to be fatigued, and an early warning signal is sent through the terminal device.
[0054] This embodiment constructs a rapid pre-job status detection method targeting the volatile psychological state. Based on the psychological state questionnaire responses from the target personnel before starting work, the method analyzes the psychological state of the target personnel and provides early warnings for those with abnormal psychological states. Secondly, during operation, eye-tracking data and facial expression images of the on-duty personnel are collected through terminal devices. Information fusion technology and multi-model result coupling methods are used, combined with deep learning algorithms, to comprehensively evaluate the on-duty status of the target personnel, achieving dynamic monitoring of on-duty personnel status and workload. This application provides a systematic status monitoring and early warning method for the target personnel, enabling timely detection and intervention of adverse conditions among on-duty personnel to reduce human-caused accidents. This improves monitoring efficiency while avoiding the problem of delayed early warnings, thus contributing to the safe operation of railway stations.
[0055] In one embodiment, such as Figure 3 As shown, in step S203 above, constructing a facial expression shape model based on the facial expression image may include the following steps:
[0056] Step S301: In the facial expression image, estimate the position of facial feature parts with the top of the head as the reference point, and uniformly set marker points on the contour of each facial feature part.
[0057] Step S302: Fit the central axis of the facial expression image based on the midpoint of the line connecting the eyebrows, the two pupils, and the center of the mouth, and divide the facial expression image into two symmetrical parts based on the central axis.
[0058] Step S303: Adjust the marker points that are symmetrical about the central axis to the same horizontal line, and construct a facial expression shape model based on the adjusted marker points.
[0059] Specifically, the server reads facial expression images, estimates the approximate location of facial features using the top of the head as a reference point, and evenly sets marker points on the contours of each facial feature. The face is divided into two symmetrical parts by the central axis of the facial expression image fitted by three points: the center of the eyebrows, the midpoint of the line connecting the two pupils, and the center of the mouth. Without scaling, translation, or rotation, the facial expression image is adjusted so that the marker points symmetrical about the central axis are aligned to the same horizontal line, and a facial expression shape model is constructed based on the adjusted marker points.
[0060] In one embodiment, step S203 above, based on extracting facial expression feature vectors through a facial expression shape model, may include the following steps:
[0061] In the facial expression shape model, multiple different feature candidate regions are divided according to the left eye and left eyebrow, the right eye and right eyebrow, and the mouth. For each feature candidate region, the difference image method is used to perform difference operations on all image sequences in the facial expression image and neutral expression images in the database to obtain the mean difference value corresponding to each feature candidate region. The facial expression feature vector is extracted from the image sequence with the largest mean difference value in each feature candidate region.
[0062] Among them, the differential image method is a technique commonly used in image processing and computer vision. It can be used to analyze changes or movements between facial expression images. This method extracts the changing parts of an image by calculating the pixel differences between two or more images.
[0063] Specifically, the server divides the facial expression shape model into multiple different regions according to the left eye and left eyebrow, the right eye and right eyebrow, and the mouth, and defines these regions as feature candidate regions. For each feature candidate region, the difference image method is used to perform difference operations on all image sequences in the facial expression image and neutral expression images in the database to obtain the mean difference value corresponding to each feature candidate region, and the facial expression feature vector is extracted from the image sequence with the largest mean difference value in each feature candidate region.
[0064] In one embodiment, step S203 above, generating an eye trajectory map and an eye movement heatmap based on eye movement data, may include the following steps:
[0065] The eye-tracking data is classified and feature-extracted according to eye-tracking trajectory, fixation time, and fixation frequency to obtain eye-tracking trajectory feature vector, fixation time feature vector, and fixation frequency feature vector. Eye-tracking analysis is performed on the eye-tracking trajectory feature vector, and an eye trajectory map is generated based on the analysis results. Based on the fixation time feature vector and fixation frequency feature vector, an eye-tracking heatmap is drawn.
[0066] It should be noted that eye trajectory maps and eye heatmaps are two commonly used representations in eye-tracking technology. Eye trajectory maps show the movement path of the eyes over a period of time, reflecting the order, speed, and dwell points of the gaze when the subject is watching or reading. Eye heatmaps are a visualization of eye movement data, which represents the frequency and time distribution of eye dwells with different color intensities, usually using color gradients to show the density of gaze.
[0067] Specifically, the server extracts features from the eye-tracking data according to eye trajectory, fixation time, and fixation frequency, resulting in three types of eye-tracking feature vectors: eye trajectory feature vector, fixation time feature vector, and fixation frequency feature vector. Eye-tracking analysis is performed based on the eye trajectory feature vector, and an eye trajectory map is generated based on the analysis results. Based on the fixation time feature vector and fixation frequency feature vector, an eye-tracking heatmap is drawn using an eye-tracking software platform.
[0068] In one embodiment, step S203 above, analyzing the psychological state of the target subject based on the psychological state questionnaire, may include the following steps:
[0069] The psychological state questionnaire is used to analyze the personality, emotional state, cognitive state, and psychological stress of the test subjects. Based on these factors, the analysis results of the psychological state are obtained. Specifically, the server analyzes the personality assessment section of the psychological state questionnaire, the emotional assessment section, the cognitive assessment section, and the psychological stress assessment section. Then, a summary analysis is performed based on these factors to obtain the final psychological state analysis results.
[0070] As an example, the server can analyze the personality traits of the target subject based on the Eysenck Personality Questionnaire and the Carter Personality Questionnaire, obtaining a personality assessment value; analyze the emotional state of the target subject based on the Self-Rating Depression Questionnaire and the State-Trait Anxiety Questionnaire, obtaining an emotional assessment value; analyze the cognitive state of the target subject based on the Wisconsin Card Sorting Questionnaire and the Mental Cognitive Ability Questionnaire, obtaining a cognitive assessment value; and analyze the psychological stress of the target subject based on the Coping Style Questionnaire and the Social Support Rating Questionnaire, obtaining a psychological stress assessment value.
[0071] In one embodiment, the method of this application further includes the following steps:
[0072] Based on preset threshold conditions, it is determined whether the personality, emotional, cognitive, and psychological stress conditions are abnormal. If at least one of these conditions is abnormal, it is determined that the psychological state of the target subject is abnormal.
[0073] The preset threshold conditions include assessment thresholds corresponding to personality traits, emotional state, cognitive state, and psychological stress, respectively.
[0074] Specifically, the server determines whether the personality, emotional, cognitive, and psychological stress conditions of the target subject are abnormal based on the personality assessment value, emotional assessment value, cognitive assessment value, psychological stress assessment value, and preset threshold conditions. If at least one of the personality, emotional, cognitive, and psychological stress conditions is identified as abnormal, it is determined that the psychological state of the target subject is abnormal.
[0075] In one embodiment, such as Figure 4 As shown, a method for monitoring and warning the state of a target object is provided in a specific embodiment, which includes the following steps:
[0076] Step S401: Obtain the psychological status questionnaire of the target subject before starting work, and analyze the psychological status of the target subject based on the psychological status questionnaire.
[0077] Step S402: Based on preset threshold conditions, determine whether the personality condition, emotional condition, cognitive condition, and psychological stress condition are abnormal; if at least one of the personality condition, emotional condition, cognitive condition, and psychological stress condition is abnormal, then determine that the psychological state of the target subject is abnormal.
[0078] Step S403: If there is an abnormality in the psychological state, an early warning message is generated according to the abnormality type corresponding to the abnormality; if there is no abnormality in the psychological state, the target to be tested is identified as the target on duty.
[0079] Step S404: Obtain eye movement data and facial expression images collected by the on-duty target object through the terminal device during the operation. Extract features from the eye movement data according to eye movement trajectory, fixation time, and fixation frequency to obtain eye movement trajectory feature vector, fixation time feature vector, and fixation frequency feature vector. Perform eye movement tracking analysis on the eye movement trajectory feature vector and generate an eye trajectory map based on the analysis results. Draw an eye movement heatmap based on the fixation time feature vector and fixation frequency feature vector.
[0080] Step S405: In the facial expression image, estimate the position of facial feature parts with the top of the head as the reference point, and uniformly set marker points on the contour of each facial feature part; fit the central axis of the facial expression image based on the midpoint of the line connecting the eyebrows, the two pupils, and the center of the mouth, and divide the facial expression image into two symmetrical parts based on the central axis; adjust the marker points symmetrical with respect to the central axis to the same horizontal line, and construct a facial expression shape model based on the adjusted marker points.
[0081] Step S406: In the facial expression shape model, the left eye and left eyebrow, right eye and right eyebrow, and mouth are divided into multiple different feature candidate regions. For each feature candidate region, the difference image method is used to perform difference operations on all image sequences in the facial expression image and neutral expression images in the database to obtain the mean difference value corresponding to each feature candidate region. The facial expression feature vector is extracted from the image sequence with the largest mean difference value in each feature candidate region.
[0082] Step S407: Based on the eye trajectory map, eye movement heat map, and facial expression feature vector, the on-duty status of the target object is comprehensively evaluated using a weighted algorithm. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0083] The beneficial effects of the above embodiments are as follows:
[0084] This application provides a systematic method for monitoring and early warning of the status of station operators. Combining human factors engineering methods, it constructs a rapid pre-job status detection method to address the short-term fluctuations in psychological state. Based on the psychological state questionnaire responses from station operators before they start work, the method analyzes the psychological state of the target personnel and provides early warnings and interventions for those with potential operational safety adaptability hazards. Secondly, during the operation of station operators, wearable devices and peripheral devices are used to collect real-time data such as eye movement and facial expression images of the workers. Information fusion technology and multi-model result coupling methods are employed, combined with deep learning algorithms, to comprehensively evaluate the on-duty status of the target personnel. This enables dynamic monitoring of on-duty personnel status and workload, allowing for timely detection and intervention of adverse conditions to reduce human-caused accidents. This improves monitoring efficiency while avoiding the problem of delayed early warnings, thus contributing to the safe operation of railway stations.
[0085] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0086] Based on the same inventive concept, this application also provides a target object state monitoring and early warning device for implementing the target object state monitoring and early warning method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more target object state monitoring and early warning device embodiments provided below can be found in the limitations of the target object state monitoring and early warning method described above, and will not be repeated here.
[0087] In one exemplary embodiment, such as Figure 5 As shown, a target object status monitoring and early warning device is provided, which may include:
[0088] The status analysis module 501 is used to obtain the psychological status questionnaire feedback of the target subject before taking up the job, and to analyze the psychological status of the target subject based on the psychological status questionnaire.
[0089] The first early warning module 502 is used to generate early warning information according to the abnormality type corresponding to the abnormality if there is an abnormality in the psychological state; if there is no abnormality in the psychological state, the target object to be tested is identified as the target object on duty.
[0090] The feature extraction module 503 is used to acquire eye movement data and facial expression images collected by the on-duty target object through the terminal device during the operation, generate eye trajectory map and eye movement heat map based on eye movement data, construct facial expression shape model based on facial expression image, and extract facial expression feature vector through facial expression shape model.
[0091] The second early warning module 504 is used to comprehensively evaluate the on-duty status of the target object based on the eye trajectory map, eye movement heat map and facial expression feature vector through a weighted algorithm. If the on-duty status is fatigued, an early warning signal is sent through the terminal device.
[0092] In one embodiment, the feature extraction module 503 is further configured to estimate the position of facial feature parts in the facial expression image with the top of the head as the reference point, and uniformly set marker points on the contour of each facial feature part; fit the central axis of the facial expression image based on the midpoint of the line connecting the eyebrows, the two pupils and the center of the mouth, and divide the facial expression image into two symmetrical parts based on the central axis; adjust the marker points symmetrical with respect to the central axis to the same horizontal line, and construct a facial expression shape model based on the adjusted marker points.
[0093] In one embodiment, the feature extraction module 503 is further configured to divide the facial expression shape model into multiple different feature candidate regions according to the left eye and left eyebrow, the right eye and right eyebrow, and the mouth; for each feature candidate region, the difference image method is used to perform difference operations on all image sequences in the facial expression image and neutral expression images in the database to obtain the mean difference value corresponding to each feature candidate region; and the facial expression feature vector is extracted from the image sequence with the largest mean difference value in each feature candidate region.
[0094] In one embodiment, the feature extraction module 503 is further configured to extract classification features from eye movement data according to eye movement trajectory, fixation time, and fixation frequency, to obtain eye movement trajectory feature vector, fixation time feature vector, and fixation frequency feature vector; perform eye movement tracking analysis on the eye movement trajectory feature vector, and generate an eye trajectory map based on the analysis results; and draw an eye movement heatmap based on the fixation time feature vector and fixation frequency feature vector.
[0095] In one embodiment, the state analysis module 501 is further configured to analyze the personality, emotional state, cognitive state, and psychological stress state of the target subject based on the psychological state questionnaire; and obtain the analysis results of the psychological state based on the personality, emotional, cognitive, and psychological stress states. In one embodiment, the device may further include: an anomaly judgment module, configured to determine whether the personality, emotional, cognitive, and psychological stress states are abnormal according to preset threshold conditions; if at least one of the personality, emotional, cognitive, and psychological stress states is abnormal, then it is determined that the psychological state of the target subject is abnormal.
[0096] Each module in the aforementioned target object status monitoring and early warning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0097] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for monitoring and providing early warning of the status of a target object.
[0098] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0099] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0100] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0101] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for condition monitoring and early warning of a target object, characterized in that, The method comprises: obtaining a psychological state questionnaire of a target object to be tested based on the feedback of a psychological state questionnaire before taking up a post, and analyzing the psychological state of the target object to be tested according to the psychological state questionnaire; if the psychological state has an abnormal condition, generating an early warning information according to the abnormal type corresponding to the abnormal condition; if the psychological state does not have an abnormal condition, determining the target object to be tested as an on-duty target object; obtaining eye movement data and facial expression images collected by a terminal device during the work process of the on-duty target object, generating an eyeball trajectory graph and an eye movement hotspot graph according to the eye movement data, constructing a facial expression shape model based on the facial expression images, and extracting a facial expression feature vector through the facial expression shape model; comprehensively evaluating the on-duty state of the on-duty target object through a weighted algorithm according to the eyeball trajectory graph, the eye movement hotspot graph and the facial expression feature vector, and if the on-duty state is a tired state, sending an early warning signal through the terminal device.
2. The method of claim 1, wherein, The method comprises: in the facial expression image, taking the top of the head as a reference point to estimate the position of the facial feature part, and uniformly setting a marker point on the contour of each facial feature part; fitting a central axis of the facial expression image according to the eyebrow center, the midpoint of the line connecting the two pupils and the center of the mouth, and dividing the facial expression image into two symmetrical parts based on the central axis; adjusting the marker points symmetrical to the central axis to the same horizontal line, and constructing the facial expression shape model based on the adjusted marker points.
3. The method of claim 2, wherein, The method comprises: in the facial expression shape model, dividing into a plurality of different feature candidate regions according to the left eye and the left eyebrow, the right eye and the right eyebrow, and the mouth; for each feature candidate region, a difference image method is used to obtain the average difference value corresponding to each feature candidate region by performing difference operation on all image sequences in the facial expression image and neutral expression images in the database; extracting the facial expression feature vector from the image sequence with the largest average difference value in each feature candidate region.
4. The method of claim 1, wherein, The method comprises: performing classification feature extraction on the eye movement data according to eye movement trajectory, fixation time and fixation times to obtain eye movement trajectory feature vector, fixation time feature vector and fixation times feature vector; performing eye movement tracking analysis on the eye movement trajectory feature vector, and generating the eyeball trajectory graph according to the analysis result; based on the fixation time feature vector and the fixation times feature vector, the eye movement hotspot graph is drawn.
5. The method according to any one of claims 1 to 4, characterized in that, The method comprises: analyzing the personality, emotion, cognition and psychological stress of the target object to be tested according to the psychological state questionnaire; based on the personality, emotion, cognition and psychological stress, the analysis result of the psychological state is obtained.
6. The method of claim 5, wherein, The method further comprises: According to a preset threshold condition, whether the personality condition, the emotion condition, the cognition condition and the psychological stress condition are abnormal is respectively determined; If at least one of the personality condition, the emotion condition, the cognition condition and the psychological stress condition is abnormal, it is determined that the psychological state of the target object to be tested has the abnormal condition.
7. A state monitoring and early warning device for a target object, characterized in that, The device comprises: a state analysis module configured to acquire a psychological state answer sheet of a target object to be tested based on a psychological state questionnaire feedback before taking up a post, and analyze the psychological state of the target object to be tested according to the psychological state answer sheet; a first early warning module configured to generate early warning information according to an abnormal type corresponding to the abnormal condition if the psychological state has the abnormal condition, and determine the target object to be tested as an on-duty target object if the psychological state does not have the abnormal condition; a feature extraction module configured to acquire eye movement data and facial expression images of the on-duty target object collected through a terminal device during a work process, generate an eye movement trajectory graph and an eye movement hotspot graph according to the eye movement data, construct a facial expression shape model based on the facial expression images, and extract a facial expression feature vector through the facial expression shape model; a second early warning module configured to comprehensively evaluate an on-duty state of the on-duty target object through a weighting algorithm according to the eye movement trajectory graph, the eye movement hotspot graph and the facial expression feature vector, and send an early warning signal through the terminal device if the on-duty state is a tired state.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6. The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6.