Visual monitoring method and system for crew workload based on voice triggering

Abstract

The invention discloses a crew workload visual monitoring method and system based on voice triggering. The method comprises the following steps: identifying load semantic information based on voice information; on the basis of triggering negative semantics in voice, starting a corresponding visual detection module, and performing visual analysis on the crew; based on a visual analysis result, carrying out comprehensive evaluation and graded early warning on the crew workload level; and according to the workload early warning level, the states of the sailors are prompted in different colors on a visual interface, and an alarm prompt is triggered. According to the method, the crew state information possibly contained in voice interaction is fully utilized, and the defect of a single information source is overcome in combination with visual analysis. According to the invention, graded early warning of the workload is realized, and a more visual and effective auxiliary decision is provided for sailor management.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method and system for visually monitoring crew workload based on voice triggering. Background Art

[0002] With the increasing complexity of modern ship operating environments and the increasing workload of crew members, crew workload management has become an important part of ensuring navigation safety and efficient ship operation. Traditional crew workload monitoring methods mainly rely on manual observation and recording, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in insufficient accuracy and timeliness of monitoring results. In addition, workload assessment systems based on physiological parameter monitoring judge their working status by measuring physiological indicators such as heart rate and blood pressure of crew members. However, such systems require crew members to wear special sensor equipment and can only reflect information at the physiological level.

[0003] Therefore, there is an urgent need for a method that can monitor crew workload without perception, so as to improve the accuracy and comprehensiveness of monitoring, timely detect and warn of excessive fatigue or stress of crew members, thereby effectively preventing safety accidents caused by excessive workload and improving the overall safety and efficiency of ship operations. Summary of the invention

[0004] In view of the defects existing in the above-mentioned prior art, the present invention provides a method for visually monitoring crew workload based on voice triggering, the method comprising:

[0005] Recognize the semantic information of the load based on the speech information;

[0006] Based on the triggering of negative semantics in the speech, the corresponding visual detection module is activated to conduct visual analysis of the crew;

[0007] Based on the visual analysis results, a comprehensive assessment and graded warning of crew workload levels are conducted;

[0008] According to the workload warning level, the crew status is prompted in different colors on the visual interface and an alarm is triggered.

[0009] Among them, natural language processing technology is used to analyze the semantic information of the text and identify sentences that express the load status.

[0010] Among them, after the load semantic information is triggered in the voice, the corresponding visual detection module is started, including starting the visual detection device at a specific position.

[0011] Wherein, the system at least comprises:

[0012] Microphone array module, which deploys multiple microphones in the cabin to collect voice data through the array;

[0013] Speech analysis equipment, which integrates speech recognition and positioning algorithms, processes data from microphone arrays in real time, identifies load-related keywords and locates the sound source that triggers keywords or semantics;

[0014] The intelligent gateway, as the central node for data interaction and control instructions, sends data to the voice analysis device and the visual analysis device or receives instructions or data from these devices;

[0015] Multiple visual analysis devices are deployed in each subspace, including cameras, responsible for capturing and analyzing the crew's visual information in real time;

[0016] The ship control center receives and comprehensively processes data from the intelligent gateway to conduct workload assessment and early warning;

[0017] An integrated microphone array, an audio codec, and an intelligent voice analysis device are arranged in the cabin, and the device communicates with other devices via Ethernet or wireless network;

[0018] At the same time, multiple intelligent visual analysis devices are arranged in the cabin, and the intelligent visual analysis devices integrate cameras, video codecs, and embedded processor components; the embedded processor runs algorithm models such as expression recognition; the devices communicate with other devices through Ethernet or other local area networks;

[0019] The system also has an intelligent gateway for coordinating the work of the voice analysis device and the visual analysis device, acting as a central node for data exchange and control instruction forwarding; it is connected to the voice analysis device and the visual analysis device via Ethernet or other local area networking; the intelligent gateway maintains a device location mapping table internally to record the location coordinates of each visual analysis device.

[0020] Among them, the voice analysis device continuously collects and analyzes the voice data in the cabin; once the keywords or semantics related to load such as fatigue and pressure are detected, the voice analysis device immediately sends a trigger message and the sound source location to the intelligent analysis gateway; the sound source location is the spatial information of the location coordinates of the sound source; the intelligent analysis gateway queries the device location mapping table to find the visual analysis device closest to the sound source location; the intelligent analysis gateway sends a startup instruction and parameter settings to the target visual analysis device; the visual analysis device starts the camera through the power control interface, and adjusts the camera's direction and field of view according to the parameter settings; the visual analysis device collects video data and runs the expression recognition algorithm; the visual analysis device sends the analysis results to the intelligent analysis gateway; the intelligent analysis gateway summarizes the results of all visual analysis devices and evaluates the comprehensive load status of the crew; the intelligent analysis gateway sends the evaluation results to the ship control center.

[0021] Among them, after triggering load-related keywords or semantics in the voice, the corresponding visual detection module is started, including calling the video data stream of the visual detection device at a specific location.

[0022] Among them, all visual detection devices in the subspace where the trigger sound source is located are called, and the visual detection device is determined based on the capture score.

[0023] wherein selecting the most appropriate visual inspection device based on the capture score includes setting a capture score priority setting;

[0024] The capture scoring process is prioritized as follows: facial capture capability, lighting adaptability, camera angle and coverage, camera resolution, and distance;

[0025] First, each visual inspection device is evaluated for its ability to capture faces. Only devices with good facial capture capabilities will enter the next step of scoring;

[0026] Facial capture capability is assessed using the following sub-items:

[0027] Field of view coverage: whether the camera's field of view covers the facial area of ​​the person where the sound source is located;

[0028] Key point detection: whether the camera supports facial key point detection and can identify changes in facial expressions;

[0029] Evaluation Methodology:

[0030] Set Boolean indicators for each vision device:

[0031]

[0032] Only when the facial capture capability = 1, the device will proceed to the next step of scoring;

[0033] On the premise that all devices meet the facial capture capability, further evaluate other factors:

[0034] Light adaptability (L): Whether the camera can clearly capture facial images under current lighting conditions;

[0035] Scoring criteria:

[0036]

[0037] Camera angle and coverage (A): Whether the camera's direction and viewing angle cover the location of the sound source;

[0038] Scoring criteria:

[0039] Scoring criteria:

[0040] Camera resolution (R): Whether the camera’s resolution can clearly capture facial expressions and movements;

[0041] Scoring criteria:

[0042] Distance (D): the distance between the camera and the sound source;

[0043] Scoring criteria:

[0044] On the basis that all devices have facial capture capabilities, the capture score of each visual inspection device is calculated through the following process:

[0045] Priority evaluation: Evaluate each indicator in order of priority, ensuring that high-level indicators are met before considering low-level indicators;

[0046] Level 1: Facial capture capabilities must be met;

[0047] Second level: Light adaptability must meet minimum requirements;

[0048] Level 3: Camera angles and coverage must meet minimum requirements;

[0049] Level 4: Camera resolution must meet minimum requirements;

[0050] Level 5: The distance between the device and the sound source must be within an acceptable range;

[0051] Each device is evaluated level by level according to priority. If the score of a certain level is lower than the minimum requirement, the device will not be selected;

[0052] That is, the minimum requirement is defined as follows: L min ≥2,A min ≥2,R min ≥2,D min ≥2,S min ≥2;

[0053] Check light adaptability: if L<2, the device is not selected;

[0054] Check the camera angle and coverage: if A<2, the device is not selected;

[0055] Check the camera resolution: if R<2, the device is not selected;

[0056] Check distance: If D<2, the device is not selected;

[0057] If there are multiple devices that meet all the minimum requirements, the total score T is calculated by weighting multiple factors for the devices that meet all the minimum requirements. The criteria are as follows:

[0058] T=w L ×L+w A ×A+w R ×R+w D ×D,

[0059] Weight sum: ∑w i =1.00;

[0060] in,

[0061] T: total score;

[0062] L: light adaptability score;

[0063] A: Camera angle and coverage score;

[0064] R: Camera resolution score;

[0065] D: distance score;

[0066] S: equipment status score;

[0067] H: historical performance score;

[0068] w L ,w A ,w R ,w D : The weight of each indicator, w i w L ,w A ,w R or D ;

[0069] Equipment selection: Sort all equipment that meets the minimum requirements by total score T from high to low;

[0070] Select the device with the highest score as the primary visual inspection device;

[0071] If higher monitoring reliability or coverage of multiple crew members is required, select the top N highest-scoring devices to work together;

[0072] Afterwards, the selected visual inspection device is called up and activated.

[0073] Among them, in the crew workload monitoring system, when the voice triggers the load-related keywords or semantics, the system starts the corresponding visual detection module to conduct a comprehensive assessment of the crew's workload level through visual data;

[0074] The system decides to initially adopt Mode 1 or Mode 2 based on the total score T of the camera, and switches the mode under certain conditions;

[0075] Among them, mode 1 is a single model selection based on high-load factors, specifically, the high-load factors are determined according to speech analysis, and the corresponding single visual analysis model is selected for evaluation;

[0076] Among them, mode 2 is a multi-model comprehensive scoring based on a probability matrix. Specifically, the probability of each high-load factor is obtained based on speech analysis, and all relevant models are called for analysis and then evaluated.

[0077] The present invention also discloses a voice-triggered visual monitoring system for crew workload, the system comprising a memory and a processor, the memory storing a computer program, and the processor being used to execute the computer program to implement any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0079] Figure 1 The flowchart shows a method for visually monitoring crew workload based on voice triggering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0082] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0083] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0084] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0085] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0086] The purpose of the present invention is to provide a method for visual monitoring of crew workload based on voice triggering, which triggers visual detection of working status through voice interaction understanding, combines voice and visual information, and more comprehensively and objectively evaluates crew workload. At the same time, it reduces the processing intensity of hardware for data calculation, reduces hardware heat dissipation, and improves service life.

[0087] like Figure 1 As shown, a method for visual monitoring of crew workload based on voice triggering includes:

[0088] Identify semantic information related to load such as fatigue and stress based on speech information.

[0089] Once load-related keywords or semantics are triggered in the voice, the corresponding visual detection module is activated to analyze the crew's facial expressions, body movements, etc. in the video data.

[0090] Based on the visual analysis results, a comprehensive assessment and graded warning of the crew's workload level are carried out.

[0091] According to the workload warning level, the crew status is prompted in different colors on the visual interface and an alarm is triggered.

[0092] The present invention provides a crew workload monitoring method using voice-triggered visual detection. The method makes full use of crew status information that may be contained in voice interaction, and combines visual analysis to overcome the shortcomings of a single information source. The present invention realizes a hierarchical early warning of workload, providing a more intuitive and effective auxiliary decision-making for crew management.

[0093] Among them, semantic information related to loads such as fatigue and stress is identified based on voice information, including real-time voice recognition and keyword extraction of voice data, or semantic understanding using natural language processing technology.

[0094] The crew’s voice data is collected in real time through a microphone array deployed in the cabin.

[0095] Perform pre-processing such as noise reduction and echo cancellation on the original voice data to improve the audio quality.

[0096] To convert the preprocessed speech data into text, large vocabulary continuous speech recognition technology based on deep learning can be used.

[0097] Match load-related keywords from the identified text, such as "sleepy", "tired", etc.

[0098] Alternatively, natural language processing technology, such as the BERT-based semantic representation model, can be used to analyze the semantic information of the text and identify sentences that express load states such as fatigue and stress.

[0099] Among them, the visual detection module includes analyzing the facial expressions of crew members and identifying negative emotional states such as fatigue and sleepiness.

[0100] In one embodiment, after triggering load-related keywords or semantics in the voice, the corresponding visual detection module is activated, including activating a visual detection device at a specific location.

[0101] Wherein, the system at least comprises:

[0102] Microphone array module, multiple microphones deployed in the cabin, collect voice data through the array.

[0103] Speech analysis equipment integrates speech recognition and positioning algorithms to process data from microphone arrays in real time, identify load-related keywords and determine the location of the sound source that triggers the keywords or semantics.

[0104] The intelligent gateway, as the central node for data interaction and control instructions, sends data to voice analysis devices and visual analysis devices or receives instructions or data from these devices.

[0105] Multiple visual analysis devices are deployed in each subspace, including cameras, which are responsible for capturing and analyzing the visual information of the crew in real time.

[0106] The ship control center receives and comprehensively processes data from the intelligent gateway to conduct workload assessment and early warning.

[0107] An integrated microphone array, an audio codec, and an intelligent voice analysis device are arranged in the cabin, and the device communicates with other devices via Ethernet or a wireless network.

[0108] At the same time, multiple intelligent visual analysis devices are arranged in the cabin, which integrate components such as cameras, video codecs, and embedded processors. The embedded processor runs algorithm models such as expression recognition. The device communicates with other devices through Ethernet or other local area networks.

[0109] The system can also be an intelligent gateway for coordinating the work of the voice analysis device and the visual analysis device, acting as a central node for data exchange and control instruction forwarding. It is connected to the voice analysis device and the visual analysis device through Ethernet or other local area networking. The intelligent gateway maintains a device location mapping table to record the location coordinates of each visual analysis device.

[0110] The voice analysis device continuously collects and analyzes voice data in the cabin. Once a keyword or semantics related to load such as fatigue or stress is detected, the voice analysis device immediately sends a trigger message and the location of the sound source to the intelligent analysis gateway. The location of the sound source can be the spatial information of the location coordinates of the sound source. The intelligent analysis gateway queries the device location mapping table to find a visual analysis device closest to the sound source location. The intelligent analysis gateway sends a startup instruction and parameter settings to the target visual analysis device. The visual analysis device starts the camera through the power control interface and adjusts the camera's direction and field of view according to the parameter settings. The visual analysis device collects video data and runs the expression recognition algorithm. The visual analysis device sends the analysis results to the intelligent analysis gateway. The intelligent analysis gateway summarizes the results of all visual analysis devices and evaluates the comprehensive load status of the crew. The intelligent analysis gateway sends the evaluation results to the ship control center.

[0111] Optionally, after triggering load-related keywords or semantics in the voice, the corresponding visual detection module is started, including calling the video data stream of the visual detection device at a specific location.

[0112] In a certain embodiment, all visual detection devices in the subspace where the trigger sound source is located are called, and the visual detection devices are determined based on the capture scores.

[0113] Wherein, the system at least comprises:

[0114] Microphone array module, multiple microphones deployed in the cabin, collect voice data through the array.

[0115] The speech analysis device integrates speech recognition and positioning algorithms to process data from the microphone array in real time, identify load-related keywords and locate the sound source.

[0116] The intelligent gateway, as the central node for data interaction and control instructions, coordinates the work of voice analysis equipment and visual analysis equipment.

[0117] Camera equipment, multiple cameras deployed in each subspace, are responsible for capturing and uploading the crew's visual information in real time.

[0118] Intelligent analysis server, responsible for centrally processing and storing data from intelligent gateways and other devices, and determining the workload of objects based on machine learning models.

[0119] The ship control center receives and comprehensively processes data from the intelligent analysis server to conduct workload assessment and early warning.

[0120] Among them, multiple microphones are arranged in the cabin to form a microphone array.

[0121] The microphone array can localize the sound source and determine the location of the crew's voice.

[0122] Connect the microphone array to the audio capture card to achieve synchronous acquisition and digitization of voice signals.

[0123] In addition, multiple cameras are arranged in the cabin to form a camera array to cover the main areas where crew activities occur.

[0124] The camera is connected to the video capture card to realize the acquisition and digitization of video signals.

[0125] And, the intelligent analysis server deploys intelligent algorithm models such as speech recognition, semantic understanding, and expression recognition. The intelligent analysis server receives the digitized audio and video data stream from the audio and video codec. When the speech analysis module detects keywords or semantics related to load such as fatigue and pressure, the visual analysis module is triggered. The intelligent analysis server sends control instructions to the camera at a specific location through the camera control interface. The camera control protocol usually adopts standard protocols such as ONVIF and RTSP. The control instructions include parameters such as camera wake-up and zoom multiple.

[0126] The microphone array continuously collects voice data in the cabin and transmits the digitized audio data stream to the intelligent analysis server through the audio codec. The speech analysis module of the intelligent analysis server performs real-time speech recognition and semantic understanding of the audio data. Once the keywords or semantics related to load such as fatigue and stress are detected, the intelligent analysis server immediately triggers the visual analysis module. The speech analysis module determines the position of the speaking crew member based on the sound source positioning results of the microphone array. The intelligent analysis server finds the best camera covering the crew position based on the preset camera position mapping table. The intelligent analysis server sends control instructions to the target camera through the camera control interface. The control module of the camera receives the control instruction, controls the target camera to complete the zoom, focuses the field of view on the crew, and transmits the collected video data stream to the intelligent analysis server. The visual analysis module of the intelligent analysis server performs expression recognition, body analysis and other processing on the video data.

[0127] In one embodiment, a microphone array is arranged in the cabin in a uniform distribution or a specific geometric layout (such as a ring or a line) to optimize the sound source localization accuracy. The microphone array continuously collects the voice information of the crew.

[0128] In one embodiment, the speech analysis device converts the preprocessed speech signal into text, and uses a continuous speech recognition technology based on deep learning (such as an end-to-end model based on a deep neural network) to extract load-related keywords (such as "sleepy" and "tired") from the recognized text, or uses natural language processing (NLP) technology to perform semantic understanding and identify sentences expressing fatigue and stress.

[0129] In one embodiment, assume that there are M microphones in the microphone array, and the positions of the microphones are P1, P2, ..., P M , the sound source position is S. The speed of sound in air is v (about 343 m / s).

[0130] Analyze the time difference Δt of each pair of microphones receiving the speech signal ij For any two microphones, i and j, calculate the distance difference between them and the sound source: Δd ij =||SP i ||-||SP j ||.

[0131] The corresponding time difference is:

[0132] The time difference Δt of each pair of microphones can be estimated by the received speech signal ij , including accurate estimation of Δt using correlation analysis or high-resolution time difference estimation algorithms such as GCCPHAT ij .

[0133] Then the time difference equations are used to solve the sound source position S.

[0134] Using the time difference information of multiple microphones, a nonlinear equation system can be established:

[0135]

[0136] The nonlinear least squares method (such as GMLS algorithm) or the positioning algorithm based on geometry (such as double difference positioning) is used to solve the sound source position S = (x s ,y s ,z s ).

[0137] In one embodiment, the cabin is divided into several subspaces (such as cockpit, engine room, living area, etc.), and the subspace to which each visual analysis device belongs and detailed location information (such as coordinates, orientation, field of view, etc.) are recorded in the device location mapping table of the intelligent gateway. All visual analysis devices located in the subspace are extracted from the device location mapping table, including information such as camera model, resolution, and viewing angle.

[0138] According to the sound source position coordinates S = (x s ,y s ,z s ), the smart gateway consults the device location mapping table to determine the specific subspace where the sound source is located.

[0139] In a certain embodiment, selecting the most suitable visual inspection device based on the capture score includes setting a capture score priority setting.

[0140] The capture scoring process is prioritized as follows: facial capture capability, lighting adaptability, camera angle and coverage, camera resolution, and distance.

[0141] Each visual inspection device is first evaluated for its ability to capture faces. This is the top priority, and only devices with good facial capture capabilities will enter the next step of scoring.

[0142] Facial capture capabilities can be assessed using the following sub-items:

[0143] Field of view coverage: Whether the camera's field of view covers the facial area of ​​the person where the sound source is located.

[0144] Key point detection: Whether the camera supports facial key point detection and can recognize changes in facial expressions.

[0145] Evaluation Methodology:

[0146] Set Boolean indicators for each vision device:

[0147]

[0148] Only when the facial capture capability = 1, the device proceeds to the next step of scoring.

[0149] On the premise that all devices meet the facial capture capability, further evaluate other factors:

[0150] 1. Light adaptability (L): Whether the camera can clearly capture facial images under the current lighting conditions.

[0151] Scoring criteria:

[0152]

[0153] 2. Camera angle and coverage (A): Whether the camera's direction and viewing angle cover the location of the sound source.

[0154] Scoring criteria:

[0155] Scoring criteria:

[0156] 3. Camera resolution (R): Is the camera resolution sufficient to clearly capture facial expressions and movements?

[0157] Scoring criteria:

[0158] 4. Distance (D): The distance between the camera and the sound source. The closer the distance, the better the capture effect.

[0159] Scoring criteria:

[0160] On the basis that all devices have facial capture capabilities, the capture score of each visual inspection device is calculated through the following process:

[0161] Priority evaluation: Evaluate each indicator in turn according to priority, ensuring that high-level indicators are met before considering lower-level indicators.

[0162] Level 1: Facial capture capabilities must be met.

[0163] Second level: Light adaptability must meet minimum requirements.

[0164] Level 3: Camera angles and coverage must meet minimum requirements.

[0165] Level 4: Camera resolution must meet minimum requirements.

[0166] Level 5: The distance between the device and the sound source must be within an acceptable range.

[0167] Each device is evaluated level by level according to priority. If the score of a certain level is lower than the minimum requirement, the device will not be selected.

[0168] That is, the minimum requirement is defined as follows: L min ≥2,A min ≥2,R min ≥2,D min ≥2,S min ≥2 Specifically,

[0169] Check light adaptability: If L<2, the device is not selected.

[0170] Check camera angle and coverage: If A<2, the device is not selected.

[0171] Check the camera resolution: If R<2, the device is not selected.

[0172] Check distance: If D<2, the device is not selected.

[0173] If there are multiple devices that meet all the minimum requirements, the total score T is calculated by weighting multiple factors for the devices that meet all the minimum requirements. The criteria are as follows:

[0174] T=w L ×L+w A ×A+w R ×R+w D ×D,

[0175] Among them, each weight w i The setting is based on the actual system requirements to ensure that more important indicators have higher weights. For example: L =0.45,w A =0.25,w R =0.15,w D =0.15, total weight:∑w i =1.00.

[0176] in,

[0177] T:Total score

[0178] L: Light adaptability score (1-3)

[0179] A: Camera angle and coverage score (1-3)

[0180] R: Camera resolution rating (13)

[0181] D: Distance score (1-3)

[0182] w L ,w A ,w R ,wD :The weight of each indicator

[0183] Equipment selection: Sort all equipment that meets the minimum requirements by total score T from high to low.

[0184] The device with the highest score is selected as the primary visual inspection device.

[0185] If higher monitoring reliability or coverage of multiple crew members is required, select the top N highest-scoring devices to work together.

[0186] After that, call and activate the selected visual inspection devices, including:

[0187] The intelligent gateway sends startup instructions and necessary parameter settings to the selected visual analysis device, such as adjusting the camera direction, field of view, etc.

[0188] After receiving the command, the selected visual analysis device starts the camera to collect real-time video data and runs the expression recognition algorithm.

[0189] The visual analysis device sends the analysis results back to the intelligent analysis server via Ethernet or wireless network.

[0190] The intelligent analysis server aggregates data from multiple visual analysis devices to evaluate and warn the crew's comprehensive workload status.

[0191] The intelligent analysis server sends the evaluation results to the ship control center.

[0192] In one embodiment, the crew issues a load-related keyword or semantics. The microphone array collects the voice signal in real time. The embedded processor performs pre-processing such as noise reduction and echo cancellation on the voice data. Speech recognition and keyword extraction are performed to identify the load-related keywords or semantics, and the sound source position S = (x s ,y s ,z s ). Once a load-related event is identified, the voice analysis device sends the trigger message and the sound source location coordinates to the intelligent gateway through the network. The intelligent gateway processes and consults the device location mapping table to determine the subspace where the sound source is located. Extract all visual analysis device information in the subspace. Send the visual analysis device information and sound source location data in the subspace to the intelligent analysis server. The intelligent analysis server receives the information of all visual analysis devices and the sound source location in the subspace. Score and filter each visual analysis device according to priority:

[0193] It includes evaluating facial capture capabilities, as well as evaluating lighting adaptability, camera angle and coverage, resolution, distance, and calculating the total score T. Select the most suitable device based on the score sorting and selection rules. Feedback the device selection results to the intelligent gateway. The intelligent gateway sends startup instructions and parameter settings (such as adjusting the camera direction, field of view, etc.) to the selected visual analysis device through the network. After receiving the instructions, the visual analysis device adjusts the camera position and starts real-time video data collection and analysis. Send the analysis results (such as facial expressions and body movements) back to the intelligent analysis server through the network.

[0194] The intelligent analysis server aggregates data from various visual analysis devices.

[0195] Combined with the voice analysis results, a comprehensive assessment is conducted to determine the crew's workload status and generate an early warning level.

[0196] The intelligent analysis server sends the evaluation results to the ship control center.

[0197] The ship control center uses different colors to indicate the crew status on the visual interface and triggers corresponding alarm prompts according to the warning level.

[0198] In one embodiment, after triggering load-related keywords or semantics in the voice, the corresponding visual detection module is started, including evaluating the crew workload level based on a specific visual detection analysis model in the visual detection device at the specific location.

[0199] Each visual analysis device is pre-installed with a fixed visual analysis model, which is specifically used to identify and evaluate the crew's fatigue, stress and other load states. The model can determine the crew's facial expression based on facial expression recognition, using a convolutional neural network (CNN) such as MTCNN detection method to detect the face position in the video frame in real time. The model is used to annotate facial key points and determine the position of eyes, mouth, eyebrows, etc. A deep learning classifier (such as a convolutional neural network-based model) is used to classify facial expressions and identify specific emotional states. For example:

[0200] Fatigue: Blinking frequency decreases and eye closure increases.

[0201] Sleepiness: droopy eyelids, yawning.

[0202] The neural network model can output the probability value of each classification result. When the probability value in any classification exceeds a specific threshold (such as 0.75), the corresponding image and classification result are reported to the ship control center, and the ship control center displays the corresponding fatigue warning information.

[0203] In a certain embodiment, after load-related keywords or semantics are triggered in the voice, the corresponding visual detection module is started, including calling the video data stream of the visual detection device at a specific location on the monitoring platform, selecting a specific visual detection analysis model to analyze the video data stream, and performing a comprehensive assessment of the crew's workload level.

[0204] In the crew workload monitoring system, when the voice triggers the load-related keywords or semantics, the system needs to start the corresponding visual detection module to comprehensively evaluate the crew's workload level through visual data. In order to take into account both the efficiency and accuracy of the system, the system decides to initially adopt mode 1 or mode 2 based on the total score T of the camera, and switches modes under specific conditions.

[0205] Among them, mode 1 is a single model selection based on high-load factors, specifically, the high-load factors are determined according to speech analysis, and the corresponding single visual analysis model is selected for evaluation. This shows that when the system confidence is high, simple and efficient evaluation methods are preferred.

[0206] Among them, mode 2 is a multi-model comprehensive scoring based on a probability matrix, which specifically obtains the probability of each high-load factor based on speech analysis, calls all relevant models for analysis, and combines the probabilities for comprehensive scoring. This indicates that when the initial mode 1 fails to confirm the high load or requires a more comprehensive evaluation, a complex multi-model analysis method is used.

[0207] In one embodiment, the initial mode selection includes:

[0208] Threshold setting: Set the total score threshold T threshold .

[0209] Judgment, if T ≥ T threshold →Select Mode 1; Else→Select Mode 2, indicating that otherwise the default mode is Mode 2.

[0210] Model 1 evaluation: The results of the single model analysis based on high-load factors are W mode1 .

[0211] Judge, if W mode1 Indicates high load → maintain mode one; Else → switch to mode two, indicating that when the judgment result of mode one is no, switch to mode two, wherein the judgment result of mode one is no, specifically the model output result of mode one is low load of crew work.

[0212] In one embodiment, the mode 1 includes identifying load-related keywords or semantics to determine high-load factors. Specific high-load factors (such as fatigue and stress) are determined based on speech analysis. According to the correspondence between the high-load factors and the analysis model, a specific visual analysis model is selected. The video data stream of the visual detection device at a specific location is called and analyzed using the selected analysis model. Based on the analysis results, the workload level of the crew is evaluated.

[0213] In one embodiment, voice trigger detection includes real-time voice data collected by a microphone array, performing pre-processing such as noise reduction and echo cancellation, and converting voice signals into text through voice recognition.

[0214] Keyword extraction or semantic understanding detects load-related keywords or sentences.

[0215] In one embodiment, in the first mode, the high load factor is determined, including:

[0216] Definition: F = {f1,f2,…,f n}: A collection of high-load factors (e.g., f1: fatigue, f2: stress).

[0217] Determine the high-load factors that trigger event associations through semantic analysis.

[0218] Output: The high-load factors involved in the current triggering event.

[0219] The corresponding relationship of model selection, each high load factor f i Corresponding to a specific visual analysis model M i :M selected ={M i |f i ∈F active}.

[0220] In one embodiment, in the mode 1, according to the location of the high-load event, the video data stream of the visual detection device at a specific location is called. i , applied to the corresponding video data stream V:R i =M i (V).

[0221] Where, V: video data stream. i : Model M i Generated analysis results. Output analysis results: R = {R i |M i ∈M selected}.

[0222] In one embodiment, mode 2 is a multi-model comprehensive evaluation method based on a probability matrix.

[0223] In one embodiment, mode 2 is a multi-model comprehensive evaluation method based on a probability matrix, which may include the following process:

[0224] Identify load-related keywords or semantics and determine the probability of each high-load factor. Based on the speech analysis results, construct a probability matrix for each high-load factor. Call all relevant visual analysis models to analyze the video data stream. Combine the probability of each factor with the analysis results, perform probability calculations, and generate multi-dimensional scores. Based on the multi-dimensional scores, evaluate the crew's workload level.

[0225] In one embodiment, in the second mode, based on the speech analysis result, a probability matrix of each high load factor is constructed, including:

[0226] definition:

[0227] P(F)={p(f1),p(f2),…,p(f n )},

[0228] Among them, p(f i ): High load factor f i The probability of existence, 0≤p(f i )≤1.

[0229] The build process includes:

[0230] Through speech analysis (based on keyword frequency, contextual semantics, etc.), for each high-load factor f i Assign a probability p(f i ), and then output the probability matrix P(F).

[0231] Model call, including definition:

[0232] M={M1,M2,…,M n}: The set of all possible visual analysis models.

[0233] The calling process specifically includes:

[0234] For each visual analysis model M i , call the video data stream V of the corresponding visual detection device for real-time analysis: R i =M i (V).

[0235] Among them, R i :Model M i The generated analysis results (such as fatigue index, stress level, etc.) are output as the analysis result set R = {R i |Mi ∈M}.

[0236] In the second mode, the probability of each factor is combined with the analysis results to perform probability calculation and generate a multi-dimensional score, including:

[0237] Probability calculation and scoring, define S = {s1, s2, ..., s n}: A collection of multi-dimensional scoring results.

[0238] Where: s i = p(f i )×R i , each score s i is the high load factor f i The probability p(f i ) and visual analysis results R i The product of .

[0239] Multi-dimensional scoring: S = {s1, s2, ..., s n}=P(F)×R, that is, multiply the corresponding elements of the probability matrix and the analysis result to obtain the score of each dimension.

[0240] The scoring formula for comprehensive workload assessment is:

[0241] By scoring all the multi-dimensional i Summarize and get the comprehensive workload assessment score W total .

[0242] In the above process, all parameter definitions include:

[0243] P(F): Probability matrix of high load factors.

[0244] p(f i ): High load factor f i The probability of existence.

[0245] M i : The i-th visual analysis model.

[0246] R i :Model M i The analysis results.

[0247] s i :Multi-dimensional scoring based on probability matrix and analysis results.

[0248] S: A set of multi-dimensional scoring results.

[0249] W total : Comprehensive workload assessment score.

[0250] In one embodiment, the system determines whether to initially adopt mode 1 or mode 2 according to the total score T of the camera, and switches the mode under certain conditions, wherein the initial mode determination includes the selection based on the total score T:

[0251] If T ≥ T threshold →Select Mode 1;

[0252] Otherwise → select mode 2;

[0253] Among them, T threshold : The total score threshold set to determine the initial mode selection.

[0254] And, mode switching under certain conditions, including:

[0255] If the result of mode 1 is negative, switch to mode 2.

[0256] That is, mode 1 evaluation: if W mode1 High load threshold not reached → switch to mode 2.

[0257] In one embodiment, the implementation process of the present invention includes:

[0258] The crew issues load-related keywords or semantics.

[0259] The microphone wind array collects voice data for preprocessing, recognition, and keyword extraction.

[0260] The sound source position S is determined by sound source localization.

[0261] The intelligent gateway determines the subspace according to S.

[0262] Extract all visual analysis devices C in the subspace.

[0263] Calculate the total score T for each visual analysis device.

[0264] Initial mode selection, judgment:

[0265] If T≥T threshold →Select Mode 1;

[0266] Else→Select mode 2;

[0267] If mode 1 is executed, determine the activated high load factor F active。 Select the corresponding visual analysis model M selected。 Call the video data stream V of the visual detection device at a specific location, and apply M selected Perform analysis. Generate analysis results R and evaluate workload V model .

[0268] Judge, if W mode1Indicates high load → Generate an alert; Else → Switch to mode 2.

[0269] After initial selection or switching, mode 2 is executed, including:

[0270] Construct the probability matrix P(F).

[0271] Call all relevant visual analysis models M.

[0272] Generate analysis result R.

[0273] Calculate the multi-dimensional score S and comprehensive score W mode2 , judgement: if W mode2 Indicates high load → generates an alert;

[0274] The comprehensive evaluation result W is sent to the ship control center.

[0275] The control center displays the crew status on a visual interface and triggers alarm prompts according to the warning level.

[0276] Use different colors (such as green, yellow, and red) to indicate crew status on the control center interface.

[0277] Depending on the warning level, the system may trigger audible and visual alarms to prompt managers to take necessary measures.

[0278] The present invention provides a crew workload monitoring method using voice-triggered visual detection. The method makes full use of crew status information that may be contained in voice interaction, and combines visual analysis to overcome the shortcomings of a single information source. The present invention realizes a hierarchical early warning of workload, providing a more intuitive and effective auxiliary decision-making for crew management.

[0279] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0280] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0281] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0282] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0283] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0284] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A method for visually monitoring crew workload based on voice triggering, the method comprising: Recognize the semantic information of the load based on the speech information; Based on the triggering of negative semantics in the speech, the corresponding visual detection module is activated to conduct visual analysis of the crew; Based on the visual analysis results, a comprehensive assessment and graded warning of crew workload levels are conducted; According to the workload warning level, the crew status is prompted in different colors on the visual interface and an alarm is triggered.

2. A method for visually monitoring crew workload based on voice triggering as claimed in claim 1, characterized in that: Natural language processing technology is used to analyze the semantic information of the text and identify sentences that express the load status.

3. A method for visually monitoring crew workload based on voice triggering as claimed in claim 1, characterized in that: After triggering the load semantic information in the voice, the corresponding visual detection module is started, including starting the visual detection device at a specific position.

4. A method for visually monitoring crew workload based on voice triggering as claimed in claim 3, characterized in that: The system at least comprises: Microphone array module, which deploys multiple microphones in the cabin to collect voice data through the array; Speech analysis equipment, which integrates speech recognition and positioning algorithms, processes data from microphone arrays in real time, identifies load-related keywords and locates the sound source that triggers keywords or semantics; The intelligent gateway, as the central node for data interaction and control instructions, sends data to the voice analysis device and the visual analysis device or receives instructions or data from these devices; Multiple visual analysis devices are deployed in each subspace, including cameras, responsible for capturing and analyzing the crew's visual information in real time; The ship control center receives and comprehensively processes data from the intelligent gateway to conduct workload assessment and early warning; An integrated microphone array, an audio codec, and an intelligent voice analysis device are arranged in the cabin, and the device communicates with other devices via Ethernet or wireless network; At the same time, multiple intelligent visual analysis devices are arranged in the cabin, and the intelligent visual analysis devices integrate cameras, video codecs, and embedded processor components; the embedded processor runs algorithm models such as expression recognition; the devices communicate with other devices through Ethernet or other local area networks; The system also has an intelligent gateway for coordinating the work of the voice analysis device and the visual analysis device, acting as a central node for data exchange and control instruction forwarding; it is connected to the voice analysis device and the visual analysis device via Ethernet or other local area networking; the intelligent gateway maintains a device location mapping table internally to record the location coordinates of each visual analysis device.

5. A method for visually monitoring crew workload based on voice triggering as claimed in claim 3, characterized in that: The voice analysis device continuously collects and analyzes the voice data in the cabin; once a keyword or semantics related to fatigue, stress, etc. is detected, the voice analysis device immediately sends a trigger message and the sound source location to the intelligent analysis gateway; the sound source location is the spatial information of the location coordinates of the sound source; The intelligent analysis gateway queries the device location mapping table to find the visual analysis device closest to the sound source; The intelligent analysis gateway sends a startup instruction and parameter settings to the target visual analysis device; the visual analysis device starts the camera through the power control interface and adjusts the camera's direction and field of view according to the parameter settings; The visual analysis device collects video data and runs the expression recognition algorithm; the visual analysis device sends the analysis results to the intelligent analysis gateway; The intelligent analysis gateway summarizes the results of all visual analysis devices and evaluates the comprehensive load status of the crew; the intelligent analysis gateway sends the evaluation results to the ship control center.

6. A method for visually monitoring crew workload based on voice triggering as claimed in claim 1, characterized in that: After triggering load-related keywords or semantics in the voice, the corresponding visual detection module is started, including calling the video data stream of the visual detection device at a specific location.

7. A method for visually monitoring crew workload based on voice triggering as claimed in claim 6, characterized in that: All visual detection devices in the subspace where the trigger sound source is located are called, and the visual detection device is determined based on the capture score.

8. A method for visually monitoring crew workload based on voice triggering as claimed in claim 7, characterized in that: selecting the most appropriate visual inspection device based on the capture score, including setting a capture score priority setting; The capture scoring process is prioritized as follows: facial capture capability, lighting adaptability, camera angle and coverage, camera resolution, and distance; First, each visual inspection device is evaluated for its ability to capture faces. Only devices with good facial capture capabilities will enter the next step of scoring; Facial capture capability is assessed using the following sub-items: Field of view coverage: whether the camera's field of view covers the facial area of ​​the person where the sound source is located; Key point detection: whether the camera supports facial key point detection and can identify changes in facial expressions; Evaluation Methodology: Set Boolean indicators for each vision device: Only when the facial capture capability = 1, the device will proceed to the next step of scoring; On the premise that all devices meet the facial capture capability, other factors are further evaluated: Light adaptability (L): whether the camera can clearly capture facial images under the current lighting conditions; Scoring criteria: Camera angle and coverage (A): Whether the camera's direction and viewing angle cover the location of the sound source; Scoring criteria: Camera resolution (R): Whether the camera’s resolution can clearly capture facial expressions and movements; Distance (D): the distance between the camera and the sound source; On the basis that all devices have facial capture capabilities, the capture score of each visual inspection device is calculated through the following process: Priority evaluation: Evaluate each indicator in order of priority, ensuring that high-level indicators are met before considering low-level indicators; Level 1: Facial capture capabilities must be met; Second level: Light adaptability must meet minimum requirements; Level 3: Camera angles and coverage must meet minimum requirements; Level 4: Camera resolution must meet minimum requirements; Level 5: The distance between the device and the sound source must be within an acceptable range; Each device is evaluated level by level according to priority. If the score of a certain level is lower than the minimum requirement, the device will not be selected; That is, the minimum requirement is defined as follows: L min ≥2,A min ≥2,R min ≥2,D min ≥2,S min ≥2; Check light adaptability: if L<2, the device is not selected; Check the camera angle and coverage: if A<2, the device is not selected; Check the camera resolution: if R<2, the device is not selected; Check distance: If D<2, the device is not selected; If there are multiple devices that meet all the minimum requirements, the total score T is calculated by weighting multiple factors for the devices that meet all the minimum requirements. The criteria are as follows: T=w L ×L+w A ×A+w R ×R+w D ×D, Weight sum: ∑w i =1.00; in, T: total score; L: light adaptability score; A: Camera angle and coverage score; R: Camera resolution score; D: distance score; w L ,w A ,w R ,w D : The weight of each indicator, w i w L ,w A ,w R or D ; Equipment selection: Sort all equipment that meets the minimum requirements by total score T from high to low; Select the device with the highest score as the primary visual inspection device; If higher monitoring reliability or coverage of multiple crew members is required, select the top N highest-scoring devices to work together; Afterwards, the selected visual inspection device is called up and activated.

9. A method for visually monitoring crew workload based on voice triggering as claimed in claim 8, characterized in that: In the crew workload monitoring system, when the voice triggers the workload-related keywords or semantics, the system starts the corresponding visual detection module to conduct a comprehensive assessment of the crew's workload level through visual data; The system decides to initially adopt Mode 1 or Mode 2 based on the total score T of the camera, and switches the mode under certain conditions; Among them, mode 1 is a single model selection based on high-load factors, specifically, the high-load factors are determined according to speech analysis, and the corresponding single visual analysis model is selected for evaluation; Among them, mode 2 is a multi-model comprehensive scoring based on a probability matrix. Specifically, the probability of each high-load factor is obtained based on speech analysis, and all relevant models are called for analysis and then evaluated.

10. A voice-triggered visual monitoring system for crew workload, the system comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 9.

Images