Monitoring System

Abstract

To provide a monitoring system capable of reducing a cost even when an image processing technique is combined with a voice processing technique and also capable of adequately monitoring an operator.SOLUTION: A monitoring system 1 includes: an image acquisition section 10 for acquiring an image related to the face of an operator of a mobile body; a voice reception section 20 for receiving voice uttered by the operator; a voice output section 40; and a control section 50. The control section 50 comprises: an image processing section 51 for processing the acquired image and also detecting an abnormality which has occurred in the operator based on an expression change of the face in the image; and a voice processing section 52 capable of achieving voice interactive processing with the operator by executing processing which includes questioning by voice to the operator and processing of the voice received in response to the questioning. The control section 50 performs control to limit the acquisition of the image in the image acquisition section 10 and control to permit the voice interactive processing on condition that the abnormality of the operator is detected in the image processing section 51.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring system that can be used in various mobile objects such as vehicles. [Background technology]

[0002] Conventionally, there has been known a vehicle control device that employs a technology also called a DMS (Driver Monitoring System) that detects abnormalities in the driver of a vehicle such as an automobile (for example, Patent Document 1). In addition, a conversational system also called a chatbot is used to enable the operation of in-vehicle devices in the vehicle to be performed in response to the utterances of the driver who is driving the vehicle. The chatbot is supposed to perform voice processing to recognize the driver's utterances. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-62911 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, there has been an increasing demand for combining DMS with voice processing technology, such as chatbots. Image processing in DMS and voice processing in chatbots both require high-load computational processing. Therefore, high-precision computing devices are required to perform these image and voice processes, resulting in high costs.

[0005] Therefore, an object of the present invention is to provide a monitoring system that can reduce costs even when image processing technology and audio processing technology are combined, and that can accurately monitor the operator. [Means for solving the problem]

[0006] (1) The monitoring system of the present invention, which is provided to solve the above-mentioned problems, comprises an image acquisition unit that acquires images related to the operator of a moving body, an audio reception unit that receives audio uttered by the operator, an audio output unit that outputs audio to the operator, and a control unit. The control unit comprises an image processing unit that can detect an abnormality occurring in the operator based on the image acquired by the image acquisition unit, and an audio processing unit that can realize audio dialogue processing for exchanging information with the operator via audio by executing processing that involves asking audio questions to the operator via the audio output unit and processing the audio received by the audio reception unit in response to the questions. The control unit is characterized in that the image processing unit controls to restrict at least a portion of the acquisition of images by the image acquisition unit and controls to allow the audio dialogue processing, on the condition that an abnormality in the operator is detected.

[0007] The above-described monitoring system is configured to restrict at least a portion of image acquisition by the image acquisition unit when an abnormality in the pilot is detected in the image processing unit. That is, once an abnormality in the pilot is detected, the above-described monitoring system restricts image acquisition and switches to voice dialogue with the pilot. Therefore, the above-described monitoring system can appropriately realize voice dialogue with the pilot while reducing the image processing burden on the image acquisition unit and image processing unit. Furthermore, because the above-described monitoring system can reduce the image processing load on the image processing unit, a high-performance processing device is not required even when both the image processing unit and the voice processing unit are combined. This allows the above-described monitoring system to be simply configured using, for example, a single microcontroller (e.g., a system-on-chip (SoC)), which is expected to reduce costs. Furthermore, the above-described monitoring system restricts image acquisition by the image acquisition unit and then allows voice dialogue processing when an abnormality in the pilot occurs, thereby preventing a loss of responsiveness in the voice dialogue processing in the voice processing unit.

[0008] (2) In the monitoring system of the present invention described above, the restriction on image acquisition by the image acquisition unit may be such that the image acquisition is stopped or the frequency of image acquisition is reduced.

[0009] By configuring the above-described monitoring system in this manner, the load of image processing can be reduced. Therefore, the versatility of the processing unit that can be used in the above-described monitoring system can be increased. As a result, the above-described monitoring system does not require high computing performance from the control unit, and therefore, cost reduction of the monitoring system can be expected. Here, the reduction in the image acquisition frequency in the image acquisition unit is performed by, for example, changing from 10 fps (initial acquisition frequency) to 1 fps (reduced acquisition frequency).

[0010] (3) In the monitoring system of the present invention described above, the image acquisition unit acquires an image of at least the face of the operator of the moving body, and the image processing unit is preferably capable of detecting an abnormality occurring in the operator from a change in facial expression of the operator contained in the image acquired by the image acquisition unit.

[0011] By configuring the above-described monitoring system as described above, it is possible to accurately grasp abnormalities in the driver based on changes in the driver's facial expression. As a result, when it is determined that the driver's facial expression has changed due to, for example, the driver dozing, yawning, or looking away, the monitoring system can transition to voice dialogue processing and engage in voice communication (dialogue) with the driver. Therefore, the monitoring system can smoothly transition to voice dialogue when an abnormality occurs in the driver's facial expression.

[0012] (4) In the monitoring system of the present invention described above, it is preferable that some or all of the restrictions on the acquisition of images by the image acquisition unit are lifted on the condition that the voice reception unit does not receive any speech from the pilot within a specified time.

[0013] By configuring the above-described monitoring system in this manner, it is possible to resume image acquisition when it is estimated that a serious abnormality has occurred in the pilot. This allows the above-described monitoring system to reliably acquire the pilot's condition while reducing the image processing load on the image processing unit. Here, examples of when the voice receiving unit does not receive a predetermined utterance from the pilot within a predetermined time include when the pilot falls asleep or when the pilot becomes unwell. The above-described predetermined time can be set to various times depending on the speed of the moving object, etc., but is preferably set to, for example, 1 to 3 seconds.

[0014] (5) In the monitoring system of the present invention described above, the frequency of image acquisition by the image acquisition unit may be varied based on an abnormal state of the operator detected by the image processing unit.

[0015] By configuring the above-described monitoring system in this manner, it is possible to acquire and process images appropriately based on the operator's abnormal state. Therefore, the above-described monitoring system can reliably detect the operator's abnormal state while reducing the load on the control unit. As a result, the above-described monitoring system does not require the control unit to have high computing performance, which is expected to reduce the cost of the monitoring system.

[0016] (6) In the monitoring system of the present invention described above, the audio processing unit may output an instruction to the audio output unit to issue an alarm if the abnormality of the pilot detected by the image processing unit is due to the pilot's posture changing by more than a predetermined range or due to the pilot dozing off for more than a predetermined period of time.

[0017] By configuring the above-described monitoring system as described above, it is possible to detect the occurrence of a serious abnormality in the operator and issue an alarm from the audio output unit. As a result, the above-described monitoring system can appropriately issue an alarm when a serious abnormality occurs in the operator. Furthermore, the above-described monitoring system can suppress the issuance of unnecessary alarms, thereby suppressing the issuance of alarms from becoming cumbersome.

[0018] (7) In the monitoring system of the present invention described above, the voice processing unit has a voice judgment unit that judges whether the voice received by the voice receiving unit contains a predetermined keyword, and the control unit controls changes to the voice output content of the voice output unit depending on the judgment result of the voice judgment unit.

[0019] By configuring the monitoring system as described above, it is possible to output appropriate voice in response to the operator's speech. Here, the predetermined keyword can be any appropriate keyword, such as "take a break," "yes," "OK," or "got it."

[0020] (8) The monitoring system of the present invention described above comprises a speed detection unit that detects the speed of the moving body, and a speed processing unit that is provided in the control unit and performs processing according to the speed detected by the speed detection unit, and the control unit may control at least one of restricting image acquisition by the image acquisition unit and changing the audio output content by the audio output unit according to the speed processing by the speed processing unit.

[0021] By configuring the above-described monitoring system in this way, it is possible to perform appropriate image processing and audio processing according to the speed of the moving object. For example, when the speed of the moving object is fast, the impact of an abnormality occurring on the operator is significant, so it is advisable to reduce the restrictions on image acquisition by the image acquisition unit and to make the audio output content from the audio output unit more appealing to the operator (e.g., an alarm). On the other hand, when the speed of the moving object is slow, it is advisable to increase the restrictions on image acquisition by the image acquisition unit and to make the audio output content from the audio output unit more gentle. [Effects of the Invention]

[0022] The present invention can provide a monitoring system that can reduce costs even when image processing technology and audio processing technology are combined, and that can accurately monitor the operator. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a configuration diagram of a monitoring system according to an embodiment of the present invention. [Figure 2] FIG. 1(a) is an explanatory diagram showing the abnormality judgment criteria (abnormality judgment A) for a pilot in a monitoring system according to one embodiment of the present invention, and FIG. 1(b) is an explanatory diagram showing the abnormality judgment criteria (abnormality judgment B) for a pilot in a monitoring system according to a modified example of the present invention. [Figure 3] 1 is an explanatory diagram showing an example of the content of audio output in the monitoring system of the present invention. FIG. [Figure 4] FIG. 1 is a flow diagram of a monitoring system of the present invention. [Figure 5] FIG. 2 is a flow diagram of audio processing (subroutine) in the monitoring system of the present invention. [Figure 6] FIG. 10 is a flow diagram of a monitoring system according to a modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] A monitoring system 1 according to one embodiment of the present invention will be described below with reference to Figures 1 and 2. In this embodiment, the moving body is an automobile (vehicle) as an example. In addition, this embodiment will be described on the assumption that a pilot (driver) is sitting in the driver's seat of the vehicle and driving the vehicle.

[0025] 1, the monitoring system 1 includes an image acquisition unit 10, an audio reception unit 20, and a speed detection unit 30. In addition to the above, the monitoring system 1 also includes an audio output unit 40, a control unit 50, etc.

[0026] The image acquisition unit 10 is configured with an imaging device such as a drive recorder or a camera (including a video camera). The image acquisition unit 10 is provided, for example, inside the vehicle cabin and can acquire an image of at least the face of the driver. The image acquisition unit 10 is preferably a camera capable of infrared photography so that it can capture an image of the driver even at night or in dark surroundings. The image acquired by the image acquisition unit 10 is processed in an image processing unit 51, which will be described later.

[0027] The voice receiving unit 20 is configured with, for example, a microphone and receives voice uttered by the driver. The voice receiving unit 20 is disposed, for example, near the steering wheel of the vehicle so as to be able to detect the voice uttered by the driver. The voice received by the voice receiving unit 20 is processed by a voice processing unit 52, which will be described later.

[0028] The speed detection unit 30 is configured with, for example, a GPS sensor. The speed detection unit 30 detects a GPS signal (speed signal) input to the GPS sensor. The speed detected by the speed detection unit 30 is processed in a speed processing unit 55, which will be described later. The GPS sensor may be one installed in an in-vehicle navigation system. Note that the speed detection unit 30 may detect the speed using a speed detection sensor installed in the vehicle instead of a GPS.

[0029] The audio output unit 40 is configured, for example, by a speaker and is arranged inside the vehicle. As will be described in detail later, the audio output unit 40 outputs audio (such as speech or an alarm) related to a predetermined question to the driver in accordance with the determination result of the driver's abnormal state.

[0030] The control unit 50 is configured, for example, with a single microcomputer (for example, an SoC: system on chip). As will be described in detail later, in this embodiment, the control unit 50 uses a processing device capable of minimum image processing and audio processing as its processing power. The control unit 50 includes an image processing unit 51, an audio processing unit 52, a speed processing unit 55, etc. The control unit 50 controls the image processing unit 51, the audio processing unit 52, and the speed processing unit 55, and can also control the entire monitoring system 1.

[0031] The image processing unit 51 processes the images acquired by the image acquisition unit 10 and can detect changes in the driver's facial expression in the processed images. The image processing unit 51 can also detect abnormalities that have occurred in the driver from changes in the driver's facial expression. In this embodiment, as shown in FIG. 2(a), an abnormality determination of the driver is made based on abnormality determination A. In abnormality determination A, a determination is made regarding changes in facial expression due to, for example, dozing, yawning, or looking away, as well as changes in driving condition that are manifested other than facial expression, such as poor posture. Abnormality determination A will be described in detail below.

[0032] "Drowsiness" is detected by detecting whether the driver's eyes are open or closed. More specifically, the image processing unit 51 determines that the driver is "drowsy" if the driver's eyes are closed for, for example, two seconds. "Yawning" is detected by detecting whether the driver's mouth is open or closed. More specifically, the image processing unit 51 determines that the driver is "yawning" if the driver's mouth is open for, for example, three seconds. "Inattentiveness" is detected by detecting feature points (e.g., six points) on the driver's face using AI and calculating the angle of the driver's face (left and right). More specifically, the image processing unit 51 determines that the driver is "looking away" if the driver's face angle is turned left or right for, for example, two seconds. "Poor posture" is detected by detecting feature points (e.g., six points) on the driver's face using AI and calculating the angle of the driver's face (up and down, roll). More specifically, the image processing unit 51 determines that the driver has "poor posture" when the angle of the driver's face is facing up or down.

[0033] 1, when it is determined that an abnormality has occurred in the driver, the image processing unit 51 performs control to restrict image acquisition by the image acquisition unit 10. Here, the restriction on image acquisition by the image acquisition unit 10 can be, for example, to stop image acquisition or to reduce the image acquisition frequency. Furthermore, the reduction in the image acquisition frequency by the image acquisition unit 10 is performed, for example, by changing from 10 fps (initial acquisition frequency) to 1 fps (reduced acquisition frequency).

[0034] The voice processing unit 52 uses AI to analyze the voice uttered by the driver received by the voice receiving unit 20, and realizes voice dialogue processing based on the driver's abnormality determination by the image processing unit 51. Specifically, the voice processing unit 52 executes processing that involves asking a voice question to the driver via the voice output unit 40 and processing the voice received by the voice receiving unit 20 in response to the question. In this way, the voice processing unit 52 realizes voice dialogue processing (for example, a function called a chatbot) that exchanges information with the driver via voice.

[0035] Here, voice dialogue processing is permitted on the condition that an abnormality in the driver is detected in the image processing unit 51. That is, in the monitoring system 1 of the present invention, when an abnormality in the driver is detected, image acquisition in the image acquisition unit 10 is restricted, and voice dialogue processing in the voice processing unit 52 is started. In other words, it is possible to prevent image processing in the image processing unit 51, which has a high load, and voice processing in the voice processing unit 52, which has a high load, from operating in parallel. Therefore, the monitoring system 1 of the present invention can reduce the load on the control unit 50.

[0036] The voice determination unit 53 determines whether the voice received by the voice receiving unit 20 includes a predetermined keyword. Any appropriate keyword can be used as the predetermined keyword, such as "take a break," "yes," "OK," or "got it." The voice determination unit 53 (control unit 50) can control changes to the voice output content in the voice output unit 40 according to the determination result of the voice determination unit 53. Details of the voice output content will be explained in the operation flow of the monitoring system 1 described later.

[0037] The speed processing unit 55 processes the speed of the vehicle detected by the speed detection unit 30. Specifically, the speed processing unit 55 classifies the speed of the vehicle detected by the speed detection unit 30 into an appropriate range (for example, 50 to 59 km / h), and can perform control to change the determination criteria (for example, the time required to determine whether the driver is dozing) in the image processing unit 51 and the audio processing unit 52 according to the classified speed range.

[0038] In this way, in this embodiment, the speed processing unit 55 (control unit 50) can control at least one of the restrictions on image acquisition in the image processing unit 51 and the changes to the audio output content in the audio output unit 40 depending on the speed.

[0039] Therefore, the above-described monitoring system 1 can perform appropriate image processing and audio processing according to the vehicle speed. For example, when the vehicle speed is high, the impact of an abnormality occurring on the driver is significant, so it is advisable to reduce the restrictions on image acquisition by the image acquisition unit 10 and to make the audio output content from the audio output unit 40 more appealing to the driver (e.g., an alarm). On the other hand, when the vehicle speed is low, it is advisable to increase the restrictions on image acquisition by the image acquisition unit 10 and to make the audio output content from the audio output unit 40 gentler, for example.

[0040] The above is the configuration of the monitoring system 1 of the present invention. Next, the operational flow of the monitoring system 1 of the present invention will be described below with reference to FIGS.

[0041] As shown in FIG. 4, when the processing of the monitoring system 1 is started, first, the image acquisition unit 10 acquires an image including at least the face of the driver (step S10).

[0042] When an image of the driver is acquired by the image acquisition unit 10, the image is processed by the image processing unit 51. As the image processing is performed by the image processing unit 51, an abnormality in the driver (pilot) is determined based on abnormality determination A (see FIG. 2(a)) (step S11).

[0043] If it is determined in step S11 that the driver is normal, the process returns to step S10 without outputting any audio (step S12).

[0044] If it is determined in step S11 that the driver is abnormal (for example, dozing off), image acquisition by the image acquisition unit 10 is stopped (step S13). When the processing in step S13 is completed, the process proceeds to a subroutine process of audio processing (step S20).

[0045] As shown in Fig. 5, when the voice processing in step S20 is started, voice is output in accordance with voice output A (see Fig. 3) (step S21). For example, if it is determined that the driver has fallen asleep, a voice saying "Thank you for driving. Are you sleepy now?" is output.

[0046] Next, the voice receiving unit 20 determines whether or not the driver has spoken (step S22). If the voice receiving unit 20 does not receive any speech from the driver within a predetermined time, the restriction on image acquisition by the image acquiring unit 10 (stopped in this embodiment) is lifted. In other words, image acquisition by the image acquiring unit 10 is resumed (step S23). Examples of cases where the voice receiving unit 20 does not receive any predetermined speech from the driver within a predetermined time include when the driver falls asleep or when the operator becomes ill. The predetermined time can be set to various times depending on the vehicle speed, etc., but is preferably set to 1 to 3 seconds, for example.

[0047] When image acquisition is resumed in step S23, a voice is output (step S24) in accordance with the voice output D. In step S24, for example, a voice saying "wake up" is output or an alarm is sounded by a buzzer or the like.

[0048] When step S24 ends, the process returns from the subroutine process in step S20 to the main routine (see FIG. 4). When the process returns to the main routine, the process returns to step S10.

[0049] As shown in Fig. 5, if it is determined in step S22 that the driver has uttered a voice within a predetermined time, a voice is output in accordance with voice output B (see Fig. 3) (step S25). For example, if it is determined that the driver has fallen asleep, a voice is output saying, "We have detected that you are falling asleep, so we are calling out to you. Why don't you take a break and have a cup of hot coffee?"

[0050] Next, the voice receiving unit 20 starts receiving voice, and it is determined whether the driver's utterance (voice) matches a predetermined keyword (step S26). In step S26, it is determined whether the driver's utterance matches a keyword (for example, "take a break," "yes," "OK," "I understand," etc.) corresponding to the content of the question from the voice processing unit 52. In this way, the monitoring system 1 described above determines whether the driver's utterance matches a keyword, and can therefore output an appropriate voice in accordance with the driver's utterance.

[0051] In step S26, if the driver's utterance does not match the predetermined keyword, a voice output is made in accordance with voice output E (see FIG. 3) (step S27). In step S26, for example, a voice output such as "Please continue to drive safely from now on." When the processing of step S27 ends, the subroutine processing of step S20 ends and the process returns to the main routine (see FIG. 4). When the process returns to the main routine, the process returns to step S10.

[0052] Furthermore, in step S26, if the driver's utterance matches a predetermined keyword, voice output is performed in accordance with voice output C (see FIG. 3) (step S28). In step S28, for example, a voice such as "Please take a good rest" is output. When the processing of step S28 ends, the subroutine processing of step S20 ends and the process returns to the main routine (see FIG. 4). When the process returns to the main routine, the process returns to step S10.

[0053] In this embodiment, after the processing in the main routine is completed, the processing returns to step S10, but the repeated processing from step S10 may be performed as needed, and the processing of the monitoring system 1 may be terminated after the processing in the main routine is completed. The above is the operational flow of the monitoring system 1 according to one embodiment of the present invention, and next, the effects of the monitoring system 1 of the present invention will be described below.

[0054] As described above, in the monitoring system 1 of the present invention, the image processing unit 51 performs control to restrict at least a portion of image acquisition by the image acquisition unit 10 on the condition that an abnormality in the operator (driver) is detected. That is, once an abnormality in the operator is detected, the monitoring system 1 described above restricts image acquisition and switches to voice dialogue with the operator. Therefore, the monitoring system 1 described above can appropriately realize voice dialogue with the operator while reducing the image processing burden in the image acquisition unit 10 and the image processing unit 51.

[0055] Furthermore, since the monitoring system 1 described above can reduce the load of image processing in the image processing unit 51, even when the image processing unit 51 and the audio processing unit 52 are both included, a high-performance arithmetic processing device is not required. As a result, the monitoring system 1 described above can be simply configured using, for example, a single microcomputer (e.g., a system-on-chip (SoC)), which is expected to reduce the cost of the monitoring system 1. Furthermore, as described above, the monitoring system 1 of the present invention does not require the control unit to have high arithmetic performance, which increases the versatility of the arithmetic processing device that can be used in the monitoring system 1. Furthermore, when an abnormality occurs with the pilot, the monitoring system 1 described above restricts image acquisition by the image acquisition unit 10 and then allows audio dialogue processing, so the responsiveness of audio dialogue processing in the audio processing unit 52 is not impaired.

[0056] Furthermore, when the monitoring system 1 detects an abnormality in the driver based on the image, it transitions to voice dialogue processing and engages in dialogue with the driver. This allows the monitoring system 1 to restore the driver's awareness and concentration on driving when there is a possibility that the driver is unable to concentrate on driving (piloting) due to drowsiness or looking away. This makes the monitoring system 1 highly effective in preventing dangers caused by drowsiness or looking away in advance.

[0057] Furthermore, the above-mentioned monitoring system 1 is configured to lift some or all of the restrictions on the acquisition of the images by the image acquisition unit 10, provided that the voice reception unit 20 does not receive any speech from the pilot within a specified period of time.

[0058] Therefore, the above-described monitoring system 1 can resume image acquisition when it is estimated that a serious abnormality has occurred in the operator. This allows the above-described monitoring system 1 to reliably acquire the operator's condition while reducing the image processing load on the image processing unit 51.

[0059] Furthermore, in the monitoring system 1 described above, the audio processing unit 52 outputs an instruction to the audio output unit 40 to issue an alarm, provided that the abnormality in the pilot detected by the image processing unit 51 is due to the pilot's posture change exceeding a predetermined range, or the pilot's dozing off for a predetermined period of time or more.

[0060] Therefore, the above-described monitoring system 1 can detect that a serious abnormality has occurred in the operator and issue an alarm from the audio output unit 40. This allows the above-described monitoring system 1 to issue an alarm appropriately when a serious abnormality has occurred in the operator. Furthermore, the above-described monitoring system 1 can suppress the issuance of unnecessary alarms, thereby preventing the issuance of alarms from becoming cumbersome.

[0061] Next, the operational flow of the monitoring system 1 according to the modified example of the present invention will be described below. Note that a description of the same content as in the above-described embodiment will be omitted.

[0062] <<Variations>> 6, when the operation of the monitoring system 1 according to the modified example starts, the image acquisition unit 10 acquires an image of the driver (step S100). The image acquisition frequency at this time is set to, for example, 10 fps.

[0063] Next, an abnormality determination of the driver is performed in accordance with abnormality determination A (see FIG. 2(a)) (step S101). If it is determined in step S101 that there is no abnormality in the driver, the process returns to step S100 without outputting any audio (step S102).

[0064] In step S101, if it is determined that the driver is abnormal (for example, dozing off at the wheel), the image acquisition frequency in the image acquisition unit 10 is reduced (step S103). The image acquisition frequency at this time is set to, for example, 1 fps. In this way, in the monitoring system 1 according to the modified example, the image acquisition frequency in the image acquisition unit 10 is changed based on the abnormal state of the driver detected by the image processing unit 51.

[0065] When the processing in step S103 is completed, the process shifts to parallel processing. In parallel processing, the processing from step S104 is performed in parallel with the audio processing (step S20) in the above-described embodiment. The audio processing (step S20) is the same as in the above-described embodiment, so a description thereof will be omitted.

[0066] In step S104, the image acquisition by the image acquisition unit 10 is performed at, for example, 1 fps.

[0067] Next, the driver's condition is determined in accordance with abnormality determination B (step S105). As shown in FIG. 2(b), abnormality determination B determines whether the driver has fallen into an abnormality related to poor posture. First, prior to abnormality determination B, the image processing unit 51 acquires the outline of the driver's face using AI, calculates the coordinates of the center position of the face from the outline of the face, and tracks (tracks the coordinates of the center position). Next, if the coordinates of the center position of the driver's face fall outside a predetermined range during tracking, it is determined that the driver has fallen into a poor posture.

[0068] As shown in FIG. 6, if it is determined in step S105 that the driver has not lost his / her posture, the process returns to step S100.

[0069] If it is determined in step S105 that the driver has lost their posture, the audio processing (step S20) being performed in parallel is stopped (step S106). When the audio processing (step S20) is stopped in step S106, audio is output in accordance with audio output D (see FIG. 3). That is, in step S106, if it is estimated that a serious abnormality has occurred that has caused the driver to lose their posture, audio output D corresponding to an alarm is output.

[0070] The above is the operational flow of the monitoring system 1 according to the modified example of the present invention, and next, the effects of the monitoring system 1 according to the modified example will be described below.

[0071] As described above, in the monitoring system 1 according to the modified example, the frequency of image acquisition by the image acquisition unit 10 varies based on the abnormal state of the operator (driver) detected by the image processing unit 51.

[0072] Therefore, the above-described monitoring system 1 can acquire and process images appropriately based on the abnormal state of the operator. As a result, the above-described monitoring system 1 can reliably detect the abnormal state of the operator while reducing the load on the control unit 50. As a result, the above-described monitoring system 1 does not require the control unit 50 to have high computing performance, and therefore, cost reduction of the monitoring system 1 can be expected.

[0073] The above is the configuration and effects of the monitoring system 1 according to the embodiment and modified examples of the present invention, but the monitoring system 1 of the present invention is not limited to the above-described embodiment and modified examples, and various modifications can be made.

[0074] In this embodiment, the control unit 50 includes the image processing unit 51, the audio processing unit 52, and the speed processing unit 55, but these processing units may be arranged separately. In addition, in this embodiment, the control unit 50 includes the speed detection unit 30 and the speed processing unit 55, but the speed detection unit 30 and the speed processing unit 55 may be provided as appropriate as necessary, and the speed detection unit 30 may not be provided. In addition, the speed detection unit 30 may use various speed detection means, such as a GPS sensor or a speed sensor mounted on the vehicle.

[0075] Furthermore, in the present embodiment, the image processing unit 51 and the audio processing unit 52 are formed as a single microcomputer (SoC), but this is not limiting. For example, the image processing unit 51 and the audio processing unit 52 may be provided independently. Furthermore, the image processing unit 51 and the audio processing unit 52 may use processing devices with various performance capabilities (including high-performance ones) taking into account the load imposed on them. The performance required for the image processing unit 51, the audio processing unit 52, etc. may be determined taking into account cost-effectiveness. The image acquisition unit 10 may use various drive recorders, cameras, video cameras, etc. capable of acquiring the driver's status. Furthermore, the image acquisition unit 10 may acquire various images capable of acquiring the driver's status instead of or in addition to facial images. For example, the image acquisition unit 10 may detect the driver's body tilt. Furthermore, the audio receiving unit 20 may use various means, such as a microphone, capable of acquiring audio. Furthermore, the audio processing unit 52 may utilize a chatbot.

[0076] Furthermore, in this embodiment, the image processing unit 51 performs control to restrict at least a portion of image acquisition by the image acquisition unit 10 on the condition that an abnormality in the pilot is detected, but this is not limited to this. The restrictions on the image acquisition unit 10 and the image processing unit 51 can be imposed using various means that can reduce the processing load on the image acquisition unit 10 and the image processing unit 51. For example, the image processing in the image processing unit 51 may be restricted. Furthermore, in this modified example, the image acquisition frequency in the image acquisition unit 10 is restricted from 10 fps to 1 fps, but the image acquisition frequency may be changed as appropriate depending on the performance of the image acquisition unit 10 to be installed.

[0077] Furthermore, the conditions for determining whether a driver is abnormal, which are detected in this embodiment, are not limited to the embodiment or the modified example, and various conditions can be set. Furthermore, the content of the sound output from the sound output unit 40 is not limited to the embodiment or the modified example, and various sound contents can be set.

[0078] In this embodiment and its modified examples, an alarm is issued if the pilot's abnormality is due to the pilot's "poor posture" or the pilot "dozing off" for a predetermined period of time or longer, but this is not limited to this and an alarm can be issued based on various criteria.

[0079] In the present embodiment and the modified examples, the voice determination unit 53 determines whether or not the voice contains a predetermined keyword, but various keywords can be used. The voice determination unit 53 can set various determination conditions that enable voice dialogue with the pilot, in addition to using keywords. For example, the voice determination unit 53 may use AI to determine the voice (utterance) of the pilot.

[0080] The above are various embodiments and modifications of the monitoring system 1 according to the present invention, but the present invention is not limited to the above-described embodiments and modifications, and it will be readily apparent to those skilled in the art that other embodiments are possible within the scope of the claims and the teachings and spirit of the present invention. [Industrial Applicability]

[0081] The monitoring system of the present invention can be used to monitor operators (drivers) of various types of mobile objects such as automobiles, airplanes, and trains. [Explanation of symbols]

[0082] 1: Monitoring system 10: Image acquisition unit 20: Voice reception section 30: Speed ​​detection unit 40: Audio output section 50: Control unit 51: Image processing unit 52: Audio processing unit 53: Audio determination unit 55: Speed ​​processing section

Claims

1. an image acquisition unit that acquires an image related to a pilot of a moving object; a voice receiving unit that receives voice uttered by the pilot; a voice output unit that outputs voice to the operator; A control unit; Equipped with The control unit an image processing unit capable of detecting an abnormality occurring in the operator based on the image acquired by the image acquisition unit; a voice processing unit capable of realizing voice dialogue processing for exchanging information with the operator via voice by executing a process involving issuing a voice question to the operator via the voice output unit and processing the voice received by the voice receiving unit in response to the question; In addition to providing A monitoring system characterized in that, when an abnormality in the pilot is detected, the image processing unit performs control to restrict at least a portion of the image acquisition by the image acquisition unit and control to allow the voice dialogue processing.

2. 2. The monitoring system according to claim 1, wherein the restriction on image acquisition by the image acquisition unit is to stop the image acquisition or to reduce the frequency of image acquisition.

3. 3. The monitoring system of claim 1 or 2, wherein the voice receiving unit releases some or all of the restrictions on the acquisition of the images by the image acquiring unit on the condition that no speech is received from the pilot within a predetermined time.

4. 3. The monitoring system according to claim 1, wherein the frequency of image acquisition by the image acquisition unit is changed based on an abnormal state of the operator detected by the image processing unit.

Images