Information processing method, device and equipment

Abstract

The invention provides an information processing method, device and equipment. The method comprises the following steps: acquiring physiological data, behavior data, voice data and environment data when a user drives or operates mechanical equipment; pre-processing the data to obtain pre-processed physiological data, pre-processed behavior data, pre-processed voice data and pre-processed environment data; obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavior data and the trained fatigue prediction model; determining a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data; determining a fatigue reminding scheme according to the fatigue monitoring result and the preprocessed environment data; and sending the fatigue reminding scheme to a control system of mechanical equipment driven or operated by the user. The method can be suitable for different application scenes such as automobiles and tower cranes, can meet the requirements of different users, is good in fatigue monitoring effect, and is beneficial to improving the safety.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue monitoring, and also relates to an information processing method, device and equipment. Background Art

[0002] In the context of intelligent production, the wide application of automation and digital technologies has greatly improved production efficiency. However, in high-intensity and high-risk working environments, the fatigue state of workers remains a key factor affecting production safety. Real-time monitoring of workers' fatigue states is particularly important for ensuring production safety and employee health. Most traditional fatigue monitoring systems rely on static rules and default configurations, and the interaction methods and response mechanisms are fixed at the system development stage. However, this design lacks real-time response capabilities, and the interface cannot automatically adjust according to users' emotions, behaviors or environmental changes, affecting the user experience; moreover, this unified design is difficult to meet the personalized needs of users in different states and environments. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an information processing method, device and equipment to meet the personalized needs of users.

[0004] To solve the above technical problems, the technical solution of the present invention is as follows:

[0005] In a first aspect of the present invention, an information processing method is provided, including:

[0006] Obtaining physiological data, behavior data, voice data and environmental data of a user when driving or operating a mechanical device;

[0007] Preprocessing the physiological data, the behavior data, the voice data and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavior data, preprocessed voice data and preprocessed environmental data;

[0008] Obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavior data and a trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to the collected historical physiological data and historical behavior data of the user;

[0009] Determining a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data;

[0010] Determining a fatigue reminder scheme according to the fatigue monitoring result and the preprocessed environmental data;

[0011] Sending the fatigue reminder scheme to the control system of the mechanical device driven or operated by the user.

[0012] Optionally, obtain the physiological data, behavioral data, voice data, and environmental data of the user when driving or operating a mechanical device, including:

[0013] Collect the physiological data of the user through a wearable device; the physiological data of the user includes heart rate data, galvanic skin response data, and body temperature data;

[0014] Collect the behavioral data of the user through an image acquisition device; the behavioral data of the user includes facial expression data, eye movement data, and head movement data;

[0015] Collect the voice data of the user through a voice acquisition device;

[0016] Collect the environmental data through a sensor; the environmental data includes temperature data and humidity data.

[0017] Optionally, preprocess the physiological data, the behavioral data, the voice data, and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data, and preprocessed environmental data, including:

[0018] Denoise the physiological data, the behavioral data, and the voice data respectively to obtain first physiological data, first behavioral data, and preprocessed voice data;

[0019] Smooth the first physiological data and the first behavioral data respectively to obtain second physiological data and preprocessed behavioral data;

[0020] Calibrate the second physiological data and the environmental data respectively to obtain preprocessed physiological data and preprocessed environmental data.

[0021] Optionally, obtain a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data, and a trained fatigue prediction model, including:

[0022] Input the preprocessed physiological data and the preprocessed behavioral data into the input layer of the fatigue prediction model to obtain a first output result;

[0023] Input the first output result into the first processing layer of the fatigue prediction model for feature extraction to obtain a second output result;

[0024] Input the second output result into the second processing layer of the fatigue prediction model for dimensionality reduction processing to obtain a third output result;

[0025] Input the third output result into the third processing layer of the fatigue prediction model for classification processing to obtain a fourth output result;

[0026] Input the fourth output result into the output layer of the fatigue prediction model for fatigue prediction to obtain a fatigue prediction result.

[0027] Optionally, the training process of the fatigue prediction model includes:

[0028] Collect the historical physiological data and historical behavior data of the user;

[0029] Perform first preprocessing on the historical physiological data to obtain preprocessed historical physiological data;

[0030] Perform second preprocessing on the historical behavior data to obtain preprocessed historical behavior data;

[0031] Perform format conversion on the preprocessed historical physiological data and the preprocessed historical behavior data to obtain historical physiological data and historical behavior data in the target format;

[0032] Extract features from the historical physiological data and the historical behavior data in the target format to obtain feature data;

[0033] Label the feature data to obtain fatigue data and non-fatigue data;

[0034] Input the fatigue data and non-fatigue data into a preset network model for training to obtain a fatigue prediction model.

[0035] Optionally, determining a fatigue monitoring result based on the fatigue prediction result and the preprocessed voice data includes:

[0036] Perform text conversion on the preprocessed voice data to obtain text data;

[0037] Perform emotion analysis based on the text data to obtain an analysis result;

[0038] Determine a fatigue monitoring result based on the analysis result and the fatigue prediction result.

[0039] Optionally, determining a fatigue reminder scheme based on the fatigue monitoring result and the preprocessed environment data includes:

[0040] Determine a fatigue prompt message based on the fatigue monitoring result;

[0041] Determine device adjustment information based on the preprocessed environment data;

[0042] Determine a fatigue reminder scheme based on the fatigue prompt message and the device adjustment information.

[0043] In the second aspect of the present invention, an information processing device is provided, including:

[0044] An acquisition module, configured to acquire physiological data, behavior data, voice data, and environmental data of a user when driving or operating a mechanical device;

[0045] A processing module, configured to respectively preprocess the physiological data, the behavior data, the voice data, and the environmental data to obtain preprocessed physiological data, preprocessed behavior data, preprocessed voice data, and preprocessed environmental data; obtain a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavior data, and a trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to collected historical physiological data and historical behavior data of a user; determine a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data; determine a fatigue reminder scheme according to the fatigue monitoring result and the preprocessed environmental data; and send the fatigue reminder scheme to a control system of a device where the user is located.

[0046] In a third aspect of the present invention, there is provided a computing device, including: a processor and a memory storing a computer program, and when the computer program is run by the processor, the method described in the first aspect is executed.

[0047] In a fourth aspect of the present invention, there is provided a computer-readable storage medium storing instructions, and when the instructions are run on a computer, the computer is caused to execute the method described in the first aspect.

[0048] The above solution of the present invention has at least the following beneficial effects:

[0049] The above solution of the present invention acquires physiological data, behavior data, voice data, and environmental data of a user when driving or operating a mechanical device, and respectively preprocesses them, and then obtains a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavior data, and a trained fatigue prediction model, and then determines a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data, and further determines a fatigue reminder scheme according to the fatigue monitoring result and the preprocessed environmental data, and finally sends the fatigue reminder scheme to a control system of the mechanical device driven or operated by the user. It is applicable to different application scenarios, such as automobiles, tower cranes, etc., can meet the needs of different users, has a good fatigue monitoring effect, and is beneficial to improving safety. Description of the Drawings

[0050] Figure 1 is a flowchart of an information processing method in an embodiment of the present invention;

[0051] Figure 2 is a structural diagram of an information processing device in an embodiment of the present invention. Detailed Embodiments

[0052] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be completely conveyed to those skilled in the art.

[0053] As Figure 1 shown, an embodiment of the present invention provides an information processing method, including the following steps:

[0054] Step 101, obtaining physiological data, behavioral data, voice data, and environmental data of a user when driving or operating a mechanical device;

[0055] Step 102, respectively preprocessing the physiological data, behavioral data, voice data, and environmental data of the user to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data, and preprocessed environmental data;

[0056] Step 103, obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data, and a trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to the collected historical physiological data and historical behavioral data of the user;

[0057] Step 104, determining a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data;

[0058] Step 105, determining a fatigue reminder plan according to the fatigue monitoring result and the preprocessed environmental data;

[0059] Step 106, sending the fatigue reminder plan to the control system of the mechanical device driven or operated by the user.

[0060] The information processing method of the embodiment of the present invention, by obtaining the physiological data, behavioral data, voice data, and environmental data of the user when driving or operating a mechanical device, and respectively preprocessing them, then obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data, and a trained fatigue prediction model, and then determining a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data, and further determining a fatigue reminder plan according to the fatigue monitoring result and the preprocessed environmental data, and finally, sending the fatigue reminder plan to the control system of the mechanical device driven or operated by the user, is applicable to different application scenarios, such as automobiles, tower cranes, etc., can meet the needs of different users, has a good fatigue monitoring effect, and is beneficial to improving safety.

[0061] Specifically, when a user is driving or operating a mechanical device, it means that the user is driving a car or an electric vehicle or operating a mechanical device such as a tower crane. The above are only examples, and it can also be a mechanical device in other scenarios.

[0062] In an alternative embodiment of the present invention, step 101 includes:

[0063] Step 1011, collecting physiological data of the user through a wearable device; the physiological data of the user includes heart rate data, skin conductance response data, and body temperature data;

[0064] Specifically, the wearable device can be a smart watch worn on the user's body or a head-mounted device. For example, a smart watch or a head-mounted device can be used to collect the user's heart rate data, skin conductance response data (data that reflects emotions and physiological states by measuring changes in skin conductance), and body temperature data. These data belong to the user's physiological data and can provide a data basis for subsequent determination of whether the user is in a fatigued state.

[0065] Step 1012, collecting behavioral data of the user through an image acquisition device; the behavioral data of the user includes facial expression data, eye movement data, and head movement data;

[0066] Specifically, the user's video or picture can be collected through a camera installed on the device where the user is located. The video or picture includes the user's facial expression data, eye movement data, and head movement data. In some specific embodiments, the user's eye movement data can also be collected through a wearable device worn on the user's body.

[0067] Step 1013, collecting voice data of the user through a voice acquisition device;

[0068] Specifically, the user's voice data can be collected through a microphone installed on the device where the user is located. By analyzing the voice data, the user's emotions, fatigue state, etc. can be obtained, providing a data basis for fatigue state monitoring and improving the accuracy of monitoring.

[0069] Step 1014, collecting environmental data through a sensor; the environmental data includes temperature data and humidity data.

[0070] Specifically, the temperature data inside / outside the device can be collected through a temperature sensor installed on the device where the user is located, and the humidity data inside / outside the device can be collected through a humidity sensor, facilitating the adjustment of appropriate temperature and humidity for the user according to the temperature data and humidity data, relieving the user's fatigue state, and improving the safety of device use.

[0071] In an alternative embodiment of the present invention, step 102 includes:

[0072] Step 1021: Denoise the physiological data, behavioral data, and speech data of the user respectively to obtain first physiological data, first behavioral data, and preprocessed speech data.

[0073] Specifically, a filter (such as a Butterworth filter) can be used to remove the noise and interference in the heart rate data of the physiological data; filter and denoise the eye movement data in the behavioral data; and denoise the speech data to remove environmental noise and background noise. Through denoising, the accuracy of various data can be improved, thereby improving the accuracy and reliability of subsequent monitoring. Here, the first physiological data includes the denoised heart rate data, skin conductance response data, and body temperature data; the first behavioral data includes facial expression data, denoised eye movement data, and also includes head movement data; the preprocessed speech data includes the denoised speech data.

[0074] Step 1022: Smooth the first physiological data and the first behavioral data respectively to obtain second physiological data and preprocessed behavioral data.

[0075] Specifically, the skin conductance response data in the first physiological data can be smoothed to remove high-frequency noise and baseline drift; the head movement data in the first behavioral data can be smoothed to remove the noise caused by small head movements. Here, the second physiological data includes the denoised heart rate data, smoothed skin conductance response data, and also includes body temperature data; the preprocessed behavioral data includes facial expression data, denoised eye movement data, and also includes smoothed head movement data.

[0076] Step 1023: Calibrate the second physiological data and the environmental data respectively to obtain preprocessed physiological data and preprocessed environmental data.

[0077] Specifically, calibrate the body temperature data in the second physiological data to ensure the accuracy of the data, and then smooth the calibrated body temperature data to remove the short-term fluctuations caused by measurement devices or environmental factors; calibrate and verify the temperature data and humidity data in the environmental data respectively to ensure the accuracy of the data, and then the calibrated environmental data can be subjected to data cleaning to remove outliers and duplicate values. Here, the preprocessed physiological data includes the denoised heart rate data, smoothed skin conductance response data, and calibrated body temperature data; the preprocessed environmental data includes the temperature data after data cleaning and the humidity data after data cleaning. By performing different preprocessing methods on different data, the accuracy of the data and the accuracy and reliability of subsequent fatigue monitoring can be improved.

[0078] In an alternative embodiment of the present invention, step 103 includes:

[0079] Step 10311: Input the preprocessed physiological data and the preprocessed behavioral data into the input layer of the fatigue prediction model to obtain a first output result.

[0080] Specifically, since the preprocessed behavioral data is image data collected by a camera, these image data can first be preprocessed, such as grayscaling, normalizing, and cropping, to highlight key feature regions (such as the eyes, mouth, etc.). Then, the preprocessed behavioral data is input into the input layer of the fatigue prediction model for format conversion to obtain dimensions (such as length and width, etc.) and resolution that can be processed by subsequent processing layers. The preprocessed physiological data is input into the input layer of the fatigue prediction model for format conversion to obtain time series data or feature vectors, etc., that can be processed by subsequent processing layers. Here, the first output result includes the format-converted behavioral data and the format-converted time series data (or feature vectors). The format-converted time series data (or feature vectors), that is, the result of the preprocessed physiological data after format conversion through the input layer. Format conversion of the preprocessed physiological data and the preprocessed behavioral data through the input layer can improve the processing efficiency of subsequent processing layers.

[0081] Step 10312: Input the first output result into the first processing layer of the fatigue prediction model for feature extraction to obtain a second output result.

[0082] Specifically, the first processing layer extracts features such as heart rate variability (HRV) and average heart rate from the heart rate data in the physiological data of the first output result, extracts the change features of skin conductance level (SCL) from the skin conductance response data, and extracts the change trend and stability features of body temperature from the body temperature data; from the facial expression data in the behavioral data of the first output result, it extracts features of key regions such as the eyes and mouth (extracted by means such as edge detection and feature point matching), extracts features such as blink frequency and eyelid closure degree from the eye movement data (which can be extracted using an eye tracking algorithm), and extracts features such as the tilt angle and movement trajectory of the head from the head movement data (which can be extracted using a head pose estimation algorithm). Here, the second output result includes feature maps of all the above-extracted features, and these features can more accurately judge the fatigue state of the user, which is beneficial to improving the accuracy of monitoring.

[0083] Step 10313: Input the second output result into the second processing layer of the fatigue prediction model for dimensionality reduction processing to obtain a third output result.

[0084] Specifically, the second processing layer mainly reduces the dimension and computational amount of the data by downsampling the second output result. The third output result can be calculated through the following formula:

[0085]

[0086] Among them, H out is the height of the feature map in the third output result, and W out is the width of the feature map in the third output result. H in is the height of the feature map in the second output result, and W in is the width of the feature map in the second output result. S is the width of the window of the second processing layer, and b is the preset step size.

[0087] Step 10314: Input the third output result into the third processing layer of the fatigue prediction model for classification processing to obtain a fourth output result;

[0088] Specifically, the third processing layer integrates the feature map in the third output result into a global feature (or a one-dimensional vector). The specific method is as follows: According to a preset traversal method (such as row-first or column-first traversal), the feature maps in the third output result are arranged in sequence into a one-dimensional vector, that is, all elements of the first row of the feature map are arranged in order, then the second row, and so on until the last row; then according to the formula y i = ∑ j (w ij ·x j ) + b i , the global feature is obtained, where y i is the output of the i-th neuron in the third processing layer, w ij is the weight between the i-th neuron and the j-th neuron in the previous layer, x j is the input of the j-th neuron in the previous layer (i.e., the element in the flattened feature vector), b i is the bias of the i-th neuron, G is the global feature, w i is the weight output by the i-th neuron, and n is the number of neurons in the third processing layer. Use a preset classification function to classify the global feature and output the probability score of the fatigue state, that is, the fourth output result.

[0089] Step 10315: Input the fourth output result into the output layer of the fatigue prediction model for fatigue prediction to obtain a fatigue prediction result.

[0090] Specifically, the output layer takes the category corresponding to the highest score in the fourth output result as the predicted fatigue state, that is, the fatigue prediction result (such as fatigue or non-fatigue, etc.).

[0091] In an optional embodiment of the present invention, the training process of the fatigue prediction model in step 103 includes:

[0092] Step 10321: Collect the historical physiological data and historical behavior data of the user;

[0093] Specifically, physiological data and behavioral data collected from a user during a past period of time when using a device (such as a car, a tower crane, etc.) can be used as the user's historical physiological data and historical behavioral data. The collection method can be to collect the user's historical physiological data through a wearable device and collect the user's historical behavioral data through a camera.

[0094] Step 10322: Perform first preprocessing on the historical physiological data to obtain preprocessed historical physiological data;

[0095] Specifically, calibrate and filter the historical physiological data to remove noise and interference, and obtain preprocessed historical physiological data.

[0096] Step 10323: Perform second preprocessing on the historical behavioral data to obtain preprocessed historical behavioral data;

[0097] Specifically, perform second preprocessing on the historical behavioral data such as grayscale conversion, normalization, cropping, etc. to highlight key feature regions (such as eyes, mouth, etc.), and obtain preprocessed historical behavioral data.

[0098] Step 10324: Perform format conversion on the preprocessed historical physiological data and the preprocessed historical behavioral data to obtain historical physiological data in a target format and historical behavioral data in a target format;

[0099] Specifically, convert the preprocessed historical physiological data and the preprocessed historical behavioral data into a format that can be accepted or processed by a preset network model to improve the training efficiency.

[0100] Step 10325: Extract features from the historical physiological data in the target format and the historical behavioral data in the target format to obtain feature data;

[0101] Specifically, heart rate variability (HRV), average heart rate, changes in skin conductance level (SCL), change trends and stability characteristics of body temperature can be extracted from the historical physiological data; features such as key regions of eyes, mouth, etc., features such as blink frequency and eyelid closure degree, and features such as head tilt angle and movement trajectory can be extracted from the historical behavioral data. The above features are used as feature data.

[0102] Step 10326: Label the feature data to obtain fatigue data and non-fatigue data;

[0103] Specifically, label each feature data as fatigue or non-fatigue. The feature data with a fatigue label is used as fatigue data, and the feature data with a non-fatigue label is used as non-fatigue data.

[0104] Step 10327: Input the fatigue data and non-fatigue data into a preset network model for training to obtain a fatigue prediction model.

[0105] Specifically, use the fatigue data and non-fatigue data to train and validate the preset network model. At the same time, the weights and bias parameters of the preset network model can be optimized in a preset manner, and methods such as cross-validation can be used to evaluate the performance of the model to avoid overfitting or underfitting, and finally obtain a fatigue prediction model.

[0106] In an optional embodiment of the present invention, step 104 includes:

[0107] Step 1041: Perform text conversion on the preprocessed speech data to obtain text data;

[0108] Specifically, a trained speech conversion model can be used to convert the preprocessed speech data into text data. Here, the training process of the speech conversion model includes:

[0109] Collect a historical speech data set; the historical speech data set includes historical speech data and its corresponding text labels;

[0110] Perform processing such as noise reduction, framing, and feature extraction on the historical speech data set to obtain training data;

[0111] Use the training data to train a preset speech network model to obtain a speech conversion model.

[0112] The process of converting the preprocessed speech data into text data includes:

[0113] First, convert the preprocessed speech data into a feature sequence;

[0114] Input the feature sequence into the speech conversion model to obtain a predicted text sequence;

[0115] Correct the predicted text sequence (such as removing unnecessary punctuation marks, word segmentation, etc.) to obtain the final text data.

[0116] Step 1042: Perform emotion analysis based on the text data to obtain an analysis result;

[0117] Specifically, according to the key features and semantic information in the text data, generate scores or probabilities for each intention category, and select the intention category with the highest probability as the final intention recognition result, that is, the analysis result. In a specific embodiment, the analysis result is that the user is fatigued.

[0118] Step 1043: Determine the fatigue monitoring result according to the analysis result and the fatigue prediction result.

[0119] Specifically, by combining the analysis results and the fatigue prediction results, the fatigue monitoring result indicating whether the user is fatigued or not fatigued can be determined. Through various means and a comprehensive analysis of whether the user is in a fatigued state, a relatively accurate fatigue monitoring result can be obtained, providing a reliable data basis for subsequent responses to the user's fatigued state and improving the effectiveness of fatigue monitoring.

[0120] In an alternative embodiment of the present invention, step 105 includes:

[0121] Step 1051, determining fatigue prompt information according to the fatigue monitoring result;

[0122] Specifically, if the fatigue monitoring result is fatigue, the corresponding fatigue prompt information may include audible and visual alarms, etc., facilitating subsequent reminders to the user based on this fatigue prompt information and alleviating the fatigued state.

[0123] Step 1052, determining device adjustment information according to the preprocessed environmental data;

[0124] Specifically, based on the temperature data and humidity data in the preprocessed environmental data, as well as the preset appropriate temperature and preset appropriate humidity data, the adjusted temperature (which can be the difference between the temperature data in the preprocessed environmental data and the preset appropriate temperature) and the adjusted humidity (which can be the difference between the humidity data in the preprocessed environmental data and the preset appropriate humidity) can be obtained. That is, the device adjustment information may include the adjusted temperature and the adjusted humidity, facilitating subsequent adjustment of the temperature and humidity inside the device by the device where the user is located according to this adjusted temperature and adjusted humidity, providing a better experience for the user.

[0125] Step 1053, determining a fatigue reminder plan according to the fatigue prompt information and the device adjustment information.

[0126] Specifically, the fatigue reminder plan includes the fatigue prompt information and the device adjustment information. The control system of the device where the user is located can remind the user and adjust the environment inside the device according to this fatigue reminder plan, improving the user experience.

[0127] In an alternative embodiment of the present invention, the device where the user is located in step 106 can be a vehicle, a tower crane, or any device in other industrial scenarios. After receiving the fatigue reminder plan, the control system of the device where the user is located can remind the user according to the fatigue prompt information such as audible and visual alarms, and take timely measures to alleviate fatigue to ensure driving or production safety. At the same time, the control system of the device where the user is located adjusts the temperature and humidity of the environment inside the device according to the device adjustment information in the fatigue reminder plan, improving the user's comfort.

[0128] A specific embodiment of the information processing method according to the embodiment of the present invention includes:

[0129] Step 111, obtain user data and environmental data;

[0130] Use wearable devices (such as smartwatches or head-mounted devices) to collect physiological data (heart rate, skin conductance response, etc.) and behavioral data (motion patterns, posture changes, etc.) of users in real time when driving or operating mechanical equipment. User data mainly includes: Visual data: The camera captures the user's facial expressions, eye states, and head movements; Voice data: The microphone records voice commands in real time and tracks the user's tone and speech rate. Environmental data can be environmental weather, internal and external device temperatures, and time information recorded by environmental sensors, etc.

[0131] Step 112, preprocess the data;

[0132] Adopt different preprocessing methods for different data. For example, perform denoising processing on heart rate data, eye movement data, and voice data; perform smoothing processing on skin conductance response data and head movement data; perform calibration and verification processing on body temperature data, temperature data, and humidity data respectively. Improve the accuracy of the data and the accuracy and reliability of subsequent fatigue monitoring.

[0133] Step 113, fatigue prediction;

[0134] Input the preprocessed physiological data and preprocessed behavioral data into the fatigue prediction model to obtain the fatigue prediction result of whether the user is in a fatigued state.

[0135] Step 114, determine the fatigue monitoring result;

[0136] Combine the emotion analysis of voice data and the fatigue prediction result to jointly determine the fatigue monitoring result of whether the user is finally fatigued or not. The analysis process is relatively comprehensive, providing a reliable data basis for subsequent responses to the user's fatigued state and improving the effectiveness of fatigue monitoring.

[0137] Step 115, determine the fatigue reminder plan;

[0138] According to the fatigue monitoring result, determine fatigue reminder information such as acoustic and optical alarms, seat vibrations, steering wheel vibrations, and screen display prompt messages; according to the preprocessed environmental data, determine device adjustment information including temperature adjustment and humidity adjustment; the fatigue reminder plan includes fatigue reminder information and device adjustment information.

[0139] Step 116, send the fatigue reminder plan to the control system of the device where the user is located.

[0140] Send the fatigue reminder plan to the device where the user is located, such as production equipment in process scenarios such as cars and tower cranes, so that the control systems of relevant devices take measures to remind the user to cope with the fatigued state and ensure the safety of driving or production.

[0141] The information processing method according to the embodiments of the present invention can adjust the output interface according to voice data to adapt to the user's state. For example, when it is detected that the user is fatigued, the output interface can be simplified to a small amount of key information to reduce interference. At the same time, a voice prompt is output to ask whether to navigate to the nearest rest area, and the night mode is switched to reduce the screen brightness. If the user is in a high-pressure state, the reminder method can be adjusted to change the voice prompt to a softer tone to avoid increasing the burden.

[0142] The following are specific application scenario examples:

[0143] 1. Fatigue monitoring and dynamic reminder

[0144] Visual monitoring: It is detected that the user yawns frequently and the eyes are closed for more than 3 seconds. Voice analysis: The user says "a little tired".

[0145] Interface response: The output interface switches to the simplified mode and asks: "Do you want to navigate to the nearest service area?"

[0146] 2. Emotion perception and interaction optimization Voice analysis: Voice recognition detects that the user is in a low mood (slow and low tone). Visual monitoring: Facial expression analysis shows an unhappy state.

[0147] Interface adjustment: Switch to the relaxing music playing interface and use a soft voice to remind the user to stay focused.

[0148] 3. Personalized content recommendation:

[0149] According to the user's historical behavior and current state, personalized content is recommended (for example, playing the user's favorite music or radio, providing corresponding entertainment options).

[0150] 4. Real-time safety prompt

[0151] By monitoring the user's fatigue and emotion, real-time safety prompts are provided. For example, when the user shows signs of fatigue, it automatically prompts "Please ensure safety and take a rest when necessary".

[0152] 5. Environmental adaptability adjustment

[0153] It can automatically adjust the comfort settings of the environment where the user is located according to changes in the internal and external working environments (such as temperature, light). For example, when the outdoor temperature is too high, it automatically lowers the indoor air conditioner temperature and adjusts the humidity, etc.

[0154] The information processing method according to the embodiments of the present invention improves the accuracy and real-time performance of fatigue monitoring, optimizes the user experience, reduces the safety risks caused by fatigue, and promotes the development of intelligent human-computer interaction technology. It can be adaptively adjusted according to different intelligent production environments and task requirements to ensure effective monitoring and improved safety in the applications of various high-risk jobs.

[0155] As Figure 2 shown in the figure, an embodiment of the present invention provides an information processing device 200, including:

[0156] An acquisition module 201, configured to acquire physiological data, behavioral data, voice data, and environmental data of a user during driving or operating a mechanical device;

[0157] A processing module 202, configured to preprocess the physiological data, the behavioral data, the voice data, and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data, and preprocessed environmental data; obtain a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data, and a trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to the collected historical physiological data and historical behavioral data of the user; determine a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data; determine a fatigue reminder plan according to the fatigue monitoring result and the preprocessed environmental data; and send the fatigue reminder plan to a control system of the mechanical device driven or operated by the user.

[0158] Optionally, acquiring physiological data, behavioral data, voice data, and environmental data of a user during driving or operating a mechanical device includes:

[0159] Acquiring physiological data of the user through a wearable device; the physiological data of the user includes heart rate data, galvanic skin response data, and body temperature data;

[0160] Acquiring behavioral data of the user through an image acquisition device; the behavioral data of the user includes facial expression data, eye movement data, and head movement data;

[0161] Acquiring voice data of the user through a voice acquisition device;

[0162] Acquiring environmental data through a sensor; the environmental data includes temperature data and humidity data.

[0163] Optionally, preprocessing the physiological data, the behavioral data, the voice data, and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data, and preprocessed environmental data includes:

[0164] Performing denoising processing on the physiological data, the behavioral data, and the voice data respectively to obtain first physiological data, first behavioral data, and preprocessed voice data;

[0165] Performing smoothing processing on the first physiological data and the first behavioral data respectively to obtain second physiological data and preprocessed behavioral data;

[0166] Calibrate the second physiological data and the environmental data respectively to obtain preprocessed physiological data and preprocessed environmental data.

[0167] Optionally, according to the preprocessed physiological data, the preprocessed behavior data, and the trained fatigue prediction model, obtain a fatigue prediction result, including:

[0168] Input the preprocessed physiological data and the preprocessed behavior data into the input layer of the fatigue prediction model to obtain a first output result;

[0169] Input the first output result into the first processing layer of the fatigue prediction model for feature extraction to obtain a second output result;

[0170] Input the second output result into the second processing layer of the fatigue prediction model for dimensionality reduction to obtain a third output result;

[0171] Input the third output result into the third processing layer of the fatigue prediction model for classification to obtain a fourth output result;

[0172] Input the fourth output result into the output layer of the fatigue prediction model for fatigue prediction to obtain a fatigue prediction result.

[0173] Optionally, the training process of the fatigue prediction model includes:

[0174] Collect the historical physiological data and historical behavior data of the user;

[0175] Perform first preprocessing on the historical physiological data to obtain preprocessed historical physiological data;

[0176] Perform second preprocessing on the historical behavior data to obtain preprocessed historical behavior data;

[0177] Perform format conversion on the preprocessed historical physiological data and the preprocessed historical behavior data to obtain historical physiological data and historical behavior data in the target format;

[0178] Perform feature extraction on the historical physiological data and the historical behavior data in the target format to obtain feature data;

[0179] Label the feature data to obtain fatigue data and non-fatigue data;

[0180] Input the fatigue data and the non-fatigue data into a preset network model for training to obtain a fatigue prediction model.

[0181] Optionally, according to the fatigue prediction result and the preprocessed speech data, determine a fatigue monitoring result, including:

[0182] Perform text conversion on the preprocessed speech data to obtain text data;

[0183] Perform emotion analysis based on the text data to obtain an analysis result;

[0184] Determine a fatigue monitoring result based on the analysis result and the fatigue prediction result.

[0185] Optionally, determine a fatigue reminder plan based on the fatigue monitoring result and the preprocessed environmental data, including:

[0186] Determine fatigue prompt information based on the fatigue monitoring result;

[0187] Determine device adjustment information based on the preprocessed environmental data;

[0188] Determine a fatigue reminder plan based on the fatigue prompt information and the device adjustment information.

[0189] The information processing device according to the embodiment of the present invention obtains physiological data, behavior data, speech data, and environmental data of a user when driving or operating a mechanical device, preprocesses them respectively, then obtains a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavior data, and a trained fatigue prediction model, and then determines a fatigue monitoring result according to the fatigue prediction result and the preprocessed speech data. Furthermore, a fatigue reminder plan is determined according to the fatigue monitoring result and the preprocessed environmental data. Finally, the fatigue reminder plan is sent to the control system of the mechanical device driven or operated by the user. It is applicable to different application scenarios, such as cars, tower cranes, etc., can meet the needs of different users, has a good fatigue monitoring effect, and is beneficial to improving safety.

[0190] It should be noted that this device is the device corresponding to the above method. All implementation manners in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effects. They will not be repeated in this embodiment.

[0191] The embodiment of the present invention further provides a computing device, including: a processor, and a memory storing a computer program. When the computer program is run by the processor, it executes the method according to any one of the above embodiments. All implementation manners in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effects. They will not be repeated in this embodiment.

[0192] An embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored. When the instructions run on a computer, the computer is caused to execute the method described in any one of the above embodiments. All implementation manners in the above method embodiments are applicable to the embodiments of this device and can also achieve the same technical effects. Details are not described again in this embodiment.

[0193] It should be noted that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations shall be regarded as equivalent solutions of the present invention. Moreover, the steps of performing the above series of processes can naturally be executed in chronological order according to the described order, but it is not necessary to be executed in chronological order. Some steps can be executed in parallel, crosswise, or independently of each other.

[0194] It should be noted that in the above embodiments, the terms "include", "comprise", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the above embodiments of the implementation manner is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described method may be executed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0195] The above is the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An information processing method, characterized in that: include: Obtaining physiological data, behavioral data, voice data, and environmental data of users when driving or operating mechanical equipment; Preprocessing the physiological data, the behavioral data, the voice data and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data and preprocessed environmental data; Obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data and the trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to the collected historical physiological data and historical behavioral data of the user; Determining a fatigue monitoring result according to the fatigue prediction result and the preprocessed voice data; Determining a fatigue reminder scheme according to the fatigue monitoring result and the pre-processed environmental data; The fatigue reminder scheme is sent to a control system of a mechanical device driven or operated by a user.

2. The information processing method according to claim 1, characterized in that: Acquire the user's physiological data, behavioral data, voice data, and environmental data when driving or operating mechanical equipment, including: Collecting physiological data of the user through a wearable device; the physiological data of the user includes heart rate data, skin electrical response data and body temperature data; Collecting user behavior data through an image acquisition device; the user behavior data includes facial expression data, eye movement data and head movement data; Collecting user's voice data through voice collection equipment; Environmental data is collected through sensors; the environmental data includes temperature data and humidity data.

3. The information processing method according to claim 1, characterized in that: Preprocessing the physiological data, the behavioral data, the voice data, and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data, and preprocessed environmental data includes: Performing denoising processing on the physiological data, the behavioral data and the voice data respectively to obtain first physiological data, first behavioral data and pre-processed voice data; performing smoothing processing on the first physiological data and the first behavioral data respectively to obtain second physiological data and preprocessed behavioral data; The second physiological data and the environmental data are calibrated to obtain pre-processed physiological data and pre-processed environmental data.

4. The information processing method according to claim 1, characterized in that: Obtaining fatigue prediction results according to the preprocessed physiological data, the preprocessed behavioral data and the trained fatigue prediction model includes: Inputting the preprocessed physiological data and the preprocessed behavioral data into an input layer of a fatigue prediction model to obtain a first output result; Inputting the first output result into a first processing layer of a fatigue prediction model for feature extraction to obtain a second output result; Inputting the second output result into the second processing layer of the fatigue prediction model for dimensionality reduction processing to obtain a third output result; Inputting the third output result into the third processing layer of the fatigue prediction model for classification processing to obtain a fourth output result; The fourth output result is input into the output layer of the fatigue prediction model to perform fatigue prediction to obtain a fatigue prediction result.

5. The information processing method according to claim 1, characterized in that: The training process of the fatigue prediction model includes: Collect users' historical physiological data and historical behavioral data; Performing a first preprocessing on the historical physiological data to obtain preprocessed historical physiological data; Performing a second preprocessing on the historical behavior data to obtain preprocessed historical behavior data; Performing format conversion on the preprocessed historical physiological data and the preprocessed historical behavioral data to obtain the historical physiological data in a target format and the historical behavioral data in a target format; Performing feature extraction on the historical physiological data in the target format and the historical behavioral data in the target format to obtain feature data; Annotate the characteristic data to obtain fatigue data and non-fatigue data; The fatigue data and non-fatigue data are input into a preset network model for training to obtain a fatigue prediction model.

6. The information processing method according to claim 1, characterized in that: Determining a fatigue monitoring result according to the fatigue prediction result and the pre-processed voice data includes: Converting the preprocessed speech data into text to obtain text data; Perform sentiment analysis based on the text data to obtain analysis results; A fatigue monitoring result is determined according to the analysis result and the fatigue prediction result.

7. The information processing method according to claim 1, characterized in that: Determining a fatigue reminder scheme according to the fatigue monitoring result and the pre-processing environment data includes: Determining fatigue prompt information according to the fatigue monitoring result; Determining equipment adjustment information according to the preprocessed environment data; A fatigue reminder scheme is determined according to the fatigue reminder information and the device adjustment information.

8. An information processing device, characterized in that: include: An acquisition module, used to acquire physiological data, behavioral data, voice data and environmental data of a user when driving or operating mechanical equipment; A processing module, used to preprocess the physiological data, the behavioral data, the voice data and the environmental data respectively to obtain preprocessed physiological data, preprocessed behavioral data, preprocessed voice data and preprocessed environmental data; Obtaining a fatigue prediction result according to the preprocessed physiological data, the preprocessed behavioral data and the trained fatigue prediction model; the fatigue prediction model is obtained by training a preset network model according to the collected historical physiological data and historical behavioral data of the user; Determine a fatigue monitoring result according to the fatigue prediction result and the pre-processed voice data; determine a fatigue reminder scheme according to the fatigue monitoring result and the pre-processed environment data; The fatigue reminder scheme is sent to the control system of the device where the user is located.

9. A computing device, characterized in that include: A processor and a memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is performed.

10. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images