Communication device and communication voice processing method

Abstract

The present invention provides a voice communication device that uses telecommunications. The voice communication device includes a microphone, a communicator that transmits and receives a call voice, a booster that boosts a high frequency level in a call voice range, a caller state collection device that collects caller state information to detect whether a caller is wearing a mask, and a processor. The processor: analyzes the caller state information to detect whether the caller is wearing a mask; outputs, to the booster, a boost control signal for controlling the high frequency boost of the call voice collected by the microphone according to the result of detecting whether the caller is wearing a mask; and causes the communicator to transmit the call voice on which the boost control process has been performed.

Description

[Technical field]

[0001] The present invention relates to a communication device and a method for processing speech sounds, and more particularly to a communication device and a method for processing speech sounds that improves the difficulty of hearing even when a person speaking is wearing a mask. [Background technology]

[0002] Proposals for improving the quality of conversations using electric communication means (hereafter, "calls") have been seen for a long time. For example, Patent Document 1 states, "Two sound quality change modes are provided: a noise mode that corrects the voice masking when the caller is in a noisy environment, and a sound quality mode that is used when the other party's voice is difficult to hear during normal calls. The data stored in the memory is read, and the digital equalizer, noise suppressor, volume, echo canceller, and sidetone adding circuit are controlled by the arithmetic processing unit using this data, making it possible to set the above-mentioned noise mode or sound quality mode. (Summary excerpt)"

[0003] In recent years, the number of occasions when people wear masks in daily life has increased to reduce the effects of hay fever and yellow dust, and to prevent viral infections such as COVID-19. One issue with wearing a mask in daily life is that it can block speech, making it difficult to hear conversations and phone calls.

[0004] Non-Patent Document 1 introduces a hearing aid that corrects the difficulty of hearing conversations caused by masks. In mask mode, this hearing aid boosts the high frequency range of sounds from 2 kHz to 5 kHz that is attenuated by the mask to correct the difficulty of hearing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2001-136239 A [Non-patent literature]

[0006] [Non-Patent Document 1] https: / / mcusercontent.com / e66304c5496b4149826f48123 / files / 4bb63ef4-8830-485b-a789-2913d8d2c6ff / Edge_Mode_Unmasking_Benefits_for_Hearing_Aid_Users.pdf?utm_source=press_media&utm_medium=pr&utm_campaign=Mask_Mode_Consumer_Campaign_2020 Summary of the Invention [Problem to be solved by the invention]

[0007] In Non-Patent Document 1, when the user judges that it is difficult to hear or when the person he / she is speaking to is wearing a mask, the mask mode can be activated. However, in a telephone conversation, the person speaking is generally far away, and it is difficult to distinguish whether the difficulty in hearing is due to the mask or the poor communication environment. From the viewpoint of smooth conversation, it is not desirable for the listener to check each time whether the speaker is wearing a mask. Therefore, the reality is that it is practically difficult for the listener to take appropriate action to alleviate the difficulty in hearing caused by the mask.

[0008] In addition, Patent Document 1 does not mention correcting the deterioration of speech quality caused by masking in the first place. Therefore, Patent Document 1 and Non-Patent Document 1 have a problem in setting the quality mode of speech sound caused by masking in a call.

[0009] The present invention has been made to solve the above-mentioned problems, and aims to correct the deterioration of speech quality caused by talking while wearing a mask, without the listener being aware of it. [Means for solving the problem]

[0010] In order to solve the above problems, the present invention has the configurations described in the claims. and others Ba, The present invention relates to A telephone device using electric communication, the telephone device comprising: a microphone; a communication device for transmitting and receiving a telephone call voice; a booster for boosting a high-frequency level in a telephone call frequency band; and a device for detecting whether a caller is wearing a mask. Imaging data Collect camera and the microphone, the communicator, the booster, and the camera a processor connected to each of the plurality of inputs, the processor comprising: The imaging data A mask detection step of analyzing and detecting whether the caller is wearing a mask, a mask detection step of executing a face recognition process on the imaging data, recognizing a caller face image in which the face of the caller is captured, and detecting whether or not the caller is wearing a mask based on whether or not a mask image in which a mask is captured can be detected from the caller face image. and, a sound attenuation estimation step of estimating a sound attenuation amount corresponding to the mask image by referring to a mask database in which a type of mask is associated with a sound attenuation amount when the mask is worn; a boost control step of outputting a boost control signal to the booster to control a high-frequency boost of the call voice collected by the microphone according to a detection result of whether the caller is wearing a mask or not; a boost control step of determining a boost amount based on the estimated audio attenuation amount and outputting the boost control signal; and a transmitting step of transmitting, from the communication device, the speech voice after the boost control process by the booster. Effect of the Invention

[0011] According to the present invention, the deterioration of speech quality caused by talking while wearing a mask can be corrected without the listener being aware of it. Note that the objects, configurations, and effects of the invention other than those described above will be made clear in the following embodiments. [Brief description of the drawings]

[0012] [Figure 1] Hardware configuration diagram of a communication device (smartphone). [Diagram 2] FIG. 1 is a functional block diagram of a program stored in a smartphone. [Diagram 3] 1 is a configuration diagram of a call system according to a first embodiment. [Figure 4] A diagram showing a home screen of a smartphone. [Diagram 5] FIG. 13 is a diagram showing a dial screen displayed in a calling program. [Figure 6] FIG. 13 shows a display screen when an incoming call is received. [Figure 7] 3 is a flowchart showing a call voice processing method according to the first embodiment. [Figure 8] FIG. 11 is a configuration diagram of a call system according to a second embodiment. [Figure 9] 10 is a flowchart showing a call voice processing method according to a second embodiment. [Figure 10] 13 is a flowchart showing a call voice processing method according to the third embodiment. [Figure 11] FIG. 13 is an explanatory diagram of classification of voice analysis processing. [Figure 12] 11 is a flowchart showing a process flow in which image analysis processing and audio analysis processing are used in combination in mask detection processing. [Figure 13] FIG. 1 is an external view of an HMD, which is one form of a communication device. [Figure 14] Hardware configuration diagram of the HMD. [Figure 15] FIG. 2 is a functional block diagram of a program stored in the HMD. [Figure 16] FIG. 1 is a diagram illustrating a call scene using an HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same configurations and steps are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.

[0014] [First embodiment] The first embodiment will be described with reference to FIG. 1 to FIG. 7. In the first embodiment, a smartphone 1 is used as an example of a call device. A camera is used as a caller status collection device for collecting caller status information to detect whether the caller is wearing a mask, and the caller status information is an embodiment using image data captured by the camera. In all embodiments, a call device using electrical communication is used, but "using electrical communication" means a communication form in which the call voice is converted into an electrical signal and transmitted / received, and the form of transmission / reception can be any type, such as wired communication, wireless communication, packet communication, or mobile communication via an IP network. In addition, in mobile communication via an IP network, a video call can be made in which video and call data consisting of call voice are sent at the same time, and the video call is also included in the call in this embodiment.

[0015] Figure 1 shows the hardware configuration of a smartphone.

[0016] The smartphone 1 includes a processor 30, a storage 40, a GPS receiver 51, a geomagnetic sensor 52, an acceleration sensor 53, a gyro sensor 54, a LAN communicator 61, a mobile communicator 62, a short-range wireless communication device 63, a display 71, an in-camera 72, an out-camera 73, a microphone 81, a speaker 82, a touch sensor 91, operation keys 92, and a booster 95, and each component is communicatively connected to each other via a system bus 31.

[0017] The processor 30 is a microprocessor unit that controls the entire smartphone 1 in accordance with a predetermined operation program, and is, for example, a CPU or a Micro Processor Unit (MPU).

[0018] The system bus 31 is a data communication path for transmitting and receiving various commands, data, and the like between the processor 30 and each component block in the smartphone 1.

[0019] The storage 40 includes a ROM 41 that stores programs for controlling the operation of the smartphone 1, a non-volatile memory 42 that stores various data such as operation setting values, detection values ​​from each sensor, objects including contents, and library information downloaded from a library, and a rewritable RAM 43 that serves as a work area used for various program operations. The non-volatile memory 42 may be a flash ROM or another memory medium.

[0020] The storage 40 needs to retain the stored information even when no power is being supplied from an external source to the smartphone 1. Therefore, instead of the non-volatile memory 42, a device such as a semiconductor element memory such as an SSD (Solid State Drive) or a magnetic disk drive such as an HDD (Hard Disk Drive) may be used.

[0021] The storage 40 can store operation programs downloaded from a network and various data created by the operation programs, etc. Also, the storage 40 can store imaging data such as moving images and still images captured using the imaging functions of the in-camera 72 and the out-camera 73.

[0022] The smartphone 1 includes a GPS (Global Positioning System) receiver 51, a geomagnetic sensor 52, an acceleration sensor 53, and a gyro sensor 54. These sensors make it possible to detect the position, inclination, direction, movement, and the like of the smartphone 1. The smartphone 1 may further include other sensors, such as an illuminance sensor, an altitude sensor, and a proximity sensor.

[0023] The LAN communication device 61 is connected to a wide area network via an access point or the like, and transmits and receives data to and from an external server on the wide area network. The connection to the access point or the like may be made by a wireless communication connection such as Wi-Fi (registered trademark).

[0024] The mobile communication device 62 performs telephone communication (calls) and data transmission and reception through wireless communication with mobile phone base stations of a mobile telephone communication network. The communication with mobile phone base stations may be performed by 4G (Generation), 5G mobile communication, W-CDMA (Wideband Code Division Multiple Access) (registered trademark) system, GSM (Global System for Mobile communications) system, LTE (Long Term Evolution) system, or other communication system. The LAN communication device 61 and the mobile communication device 62 perform voice calls through mobile communication and connect to a wide area network through wireless LAN.

[0025] The short-range wireless communication device 63 exchanges information with an external Bluetooth (registered trademark) device or an external NFC-compatible device by means of Bluetooth (registered trademark) communication, NFC standard communication, or the like.

[0026] The LAN communication device 61, the mobile communication device 62, and the short-range wireless communication device 63 each include an encoding circuit, a decoding circuit, an antenna, etc. In addition to the above-mentioned communication devices, other communication devices such as an infrared communication device may also be included.

[0027] The display 71 is a display device such as a backlit LCD display or a self-luminous organic EL display, and displays image data or video data obtained via a wide area network, or imaging data captured by the in-camera 72 or the out-camera 73.

[0028] The front camera 72 is provided on the same surface as the surface on which the display 71 of the smartphone 1 is provided. The front camera 72 captures an image of a caller (who can be both a speaker and a listener) using the smartphone 1, and generates face image data of the caller. The smartphone 1 executes face recognition processing based on this face image data, and is also used for so-called Face ID, which permits the use of the smartphone 1.

[0029] The rear camera 73 is provided on the back side of the smartphone 1. The rear camera 73 is used to capture images of scenery and the like.

[0030] Each of the in-camera 72 and the out-camera 73 is a camera that inputs image information of the surroundings and objects by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0031] The microphone 81 converts sounds in the real space, the user's voice, and the like into audio information and inputs it.

[0032] The speaker 82 outputs audio information required by the user, etc. Of course, earphones and headphones can also be connected, and it goes without saying that they can be used according to the purpose.

[0033] The touch sensor 91 is disposed in a layered manner on the display screen of the display 71 .

[0034] The operation keys 92 are configured by arranging button switches and the like.

[0035] The touch sensor 91 and the operation keys 92 are an example of an operation input device for inputting an operation instruction to the smartphone 1, and may be other operation input devices. In addition, the smartphone 1 may be operated using a separate mobile terminal device connected by wired communication or wireless communication using the LAN communication device 61 or the short-range wireless communication device 63.

[0036] Moreover, the image captured by the inner camera 72 may be analyzed, and the smartphone 1 may be operated by a gesture or other action.

[0037] FIG. 2 is a functional block diagram of a program stored in the smartphone.

[0038] The non-volatile memory 42 stores, as processing programs, a basic operation program 421 and a call program 422. The non-volatile memory 42 further includes a data storage area 430 for storing various data used when executing each program. The processor 30 reads each program stored in the non-volatile memory 42, and refers to the data stored in the data storage area 430 as necessary.

[0039] 2, when the call processing unit 4222 detects a push signal of the call application icon 22 (see FIG. 4) or an incoming call signal from another call device, the processor 30 executes the function of the mask detection unit 4220. In the first embodiment, the mask detection unit 4220 detects the presence or absence of a mask by image analysis processing, so it starts up the in-camera 72, acquires imaging data, performs image analysis processing based on the imaging data, and performs mask detection processing.

[0040] The mask detection unit 4220 outputs the mask presence / absence detection result to the boost control unit 4221. The boost control unit 4221 outputs a boost control signal according to the mask presence / absence detection result to the booster 95. The booster 95 performs high-frequency boost processing on the call voice acquired from the microphone 81, and outputs and transmits the result from the call processing unit 4222 to the communication device.

[0041] In the third embodiment, the presence or absence of a mask is detected by performing audio analysis of the call voice collected by the microphone 81, so in this case, the mask detection unit 4220 detects the presence or absence of a mask by obtaining the call voice from the microphone 81, not from the in-camera 72. Details of the movement of each signal and data shown in FIG. 2 will be made clear from time to time in the following explanation.

[0042] The mobile communication network 5 (see FIG. 3) loads a basic operation program 421 and a call program 422 into the RAM 43 and executes them.

[0043] The various data stored in the data storage area 430 are mainly data necessary for executing the call program 422. For example, there is a caller face image (called "registered face data 431") in which the face of the caller is captured, a mask database (D / B) 432, registered voice data 433, and class data 434, which are stored as necessary depending on the call voice processing method. The non-volatile memory 42 may be a single memory medium as shown in the figure, or may be composed of multiple memory media.

[0044] Fig. 3 is a configuration diagram of a call system according to the first embodiment. Callers 2 and 2A can be both speakers and listeners. A case will be described in which a call is made using mobile communication, but an Internet protocol (also referred to as an IP protocol) may be used over a wireless LAN. In Fig. 3, either the smartphone 1 or the smartphone 1A may be a call device according to the present invention, or both the smartphone 1 and the smartphone 1A may be call devices according to the present invention.

[0045] 3, reference numerals 3 and 3A denote masks, reference numerals 4 and 4A denote mobile communication base stations, reference numeral 5 denotes a mobile communication network, reference numerals 6 and 6A denote wireless LAN access points, reference numerals 60 and 60A denote wireless LAN signals, and reference numeral 7 denotes a wide area network. The mobile communication base stations 4 and 4A are main components of the mobile communication network 5, and are also components of the wide area network 7.

[0046] The caller 2 starts a call, for example by dialing up using the smartphone 1. A signal for the call is transmitted to the mobile communication base station 4, and then transmitted to the smartphone 1A held by the caller 2A via the mobile communication network 5 and the mobile communication base station 4A. When the caller 2A accepts the call on the smartphone 1A, a call is made between the callers 2 and 2A.

[0047] Meanwhile, the smartphones 1 and 1A are also connected to a wide area network 7. There are two types of paths: a path via mobile communication base stations 4 and 4A, and a path using access points 6 and 6A with wireless LAN signals 60 and 60A.

[0048] 4 to 6 show display screens of the smartphone 1. For the sake of explanation, the following will be described using the smartphone 1 operated by the caller 2 in FIG. 3 as an example, but the explanation regarding the smartphone 1 also applies to the smartphone 1A.

[0049] 4 is a diagram showing a home screen 20 of the smartphone 1. The call port 83 is connected to a microphone 81 and a speaker 82. The operation key 92 is pressed to display the home screen 20.

[0050] The home screen 20 has the time, location information, weather information, and a search window at the top, but these are not essential. What is important on the home screen 20 is the icon button group 21 for starting application programs. Pressing an icon button corresponding to an application program moves from the home screen 20 to the application program screen.

[0051] The icon button group 21 includes a call application icon 22 for starting a call program. The callers 2 and 2A press the button to start the call program.

[0052] 5 is a diagram showing a dial screen 20a displayed by a calling program. A numeric keypad 23 and a dialing start button 24 are displayed, and the caller inputs the callee's number using the numeric keypad 23 and presses the dialing start button 24 to call the callee. Another method of calling the callee may be to display a telephone directory and select the callee from the telephone directory.

[0053] Furthermore, the calling program is started in a case other than when the caller presses the call application icon 22. Figure 6 illustrates such a case, showing a display screen 20b when an incoming call is received.

[0054] When a call is received, the display screen 20b displays information 27 identifying the other party (for example, the name of the other party, the telephone number, and information indicating that the call is withheld), a call permission button 25, and a call rejection button 26. When the other party calls, the user confirms the other party's name and presses the call permission button 25 to start the call.

[0055] The dial screen 20a in Fig. 5 and Fig. 6 and the display screen 20b when a call is received correspond to the initial screen of the call program. On these initial screens, the caller looks at the display screen. At this time, the in-camera 72 is activated to capture an image of the caller's face.

[0056] This process is realized by the processor 30 activating the calling program 422 in response to detection of a pressing signal of the calling application icon 22 of the calling program, and the calling program 422 outputting an image capturing instruction signal for the in-camera 72.

[0057] 7 is a flowchart showing the call voice processing method according to the first embodiment. As described in FIG.

[0058] At the start of the process, the home screen 20 is displayed on the smartphone 1 (S10). When the caller presses down the call application icon 22 (see FIG. 4) (S11: YES), or when a call is received (see FIG. 5) (S12: YES), the processor 30 starts the call program 422. When both steps S11 and S12 are NO, the process waits in step S10.

[0059] If S11 or S12 is YES, the call program 422 executes a call process (S13 to S17) and an image analysis process (S21 to S24) in a multi-process manner.

[0060] In the image analysis process (S20), the mask detection unit 4220 activates the in-camera 72 (S21) and acquires a camera image (S22). The mask detection unit 4220 executes a subject detection process on the camera image and executes a face recognition process (S23). The mask detection unit 4220 determines whether a mask is included in the face image (S24: corresponds to a mask detection step).

[0061] If it is determined that the mask is present (S24: YES), a boost control signal indicating that a boost is to be performed is output to the boost control unit 4221. If it is determined that the mask is not present (S24: NO), a boost control signal indicating that a boost is not to be performed is output to the boost control unit 4221.

[0062] In the call process, the call processor 4222 sends a call request to the dialed number (S13), and starts the call if the call recipient gives permission (S15). Alternatively, the call processor 4222 responds to an incoming call request from the communication recipient (S14), and starts the call (S15).

[0063] The booster 95 performs boost control processing according to the boost control signal (corresponding to S16: boost control step). The booster 95 controls the amount of boost in the high frequency range of the call frequency band, specifically, 2 kHz to 5 kHz, to be ON or OFF, using the boost control signal output in step S24. The smartphone 1 transmits the call voice that has been boost-controlled in step S16 to the mobile communication network 5 or the wide area network 7 (corresponding to a transmission step).

[0064] The call processor 4222 determines whether to end the call (S17), and if YES, ends the call and returns to the home screen (S10), and if NO, continues the call.

[0065] The image analysis process S20 is performed at the start of the call program to execute mask detection, but may be continued even during the call as shown by the dashed line returning from S23 to S22 in Fig. 8. By continuing the process, a change in state, such as putting on or taking off a mask, can be detected during the call. Furthermore, in the face recognition process of S23, the first recognized face image may be stored in the data storage area 430 as registered face data 431, and the registered face data 431 may be referenced. By performing mask detection on the same face image as the registered face image, erroneous mask detection results are prevented when a face other than the caller is captured.

[0066] Although the booster 95 is realized by a hardware circuit, it can also be implemented by software processing in accordance with a program stored in the storage 40 by the processor 30 .

[0067] As described above, according to the first embodiment of the telephone device and telephone call voice processing method, during a telephone call via telecommunications, it is possible to automatically set whether or not the speaker is wearing a mask and to automatically set a high-frequency boost, thereby improving the difficulty of hearing during a telephone call without the listener having to worry about whether or not the speaker is wearing a mask.

[0068] [Second embodiment] The second embodiment will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a configuration diagram of a call system according to the second embodiment.

[0069] 8, in the second embodiment, a call service server 8 is connected to a wide area network 7. The call service server 8 includes a mask database 8a. A call is made between a caller 2 and a caller 2A, and the smartphone 1 (or the smartphone 1A) uses the mask database 8a of the call service server 8.

[0070] The mask database 8a is created by the call service server 8 by collecting mask information in the wide area network 7. The mask information includes mask images, material data, etc., and mask data with similar mask shapes and textures is selected using a mask image from among the face images of the caller captured by the camera of the smartphone 1 as a search key. The amount of attenuation of the voice when passing through the mask is estimated from information such as the material of the selected mask.

[0071] The mask database 8a may be stored in the call service server 8 and referenced, or may be downloaded in advance to the smartphone 1 and stored in the data storage area 430 of the FROM 42 as shown in FIG. 2 (see the mask database 432 in FIG. 2).

[0072] FIG. 9 is a flowchart showing a call voice processing method according to the second embodiment.

[0073] In the image analysis process S20 of the flowchart shown in FIG. 9, the mask detection unit 4220 determines whether a mask is included in the face image (S24: YES), and then refers to the mask database 432 or 8a to search for information such as the mask material (S25).

[0074] Each level of boost amount corresponds to a plurality of mask materials. The search result is a boost control signal together with no mask (S24: NO, no boost). At this time, if there is no matching material in the mask database 432 or 8a, or if the mask database 8a cannot be accessed, a default boost amount is used.

[0075] As described above, the telephone device and telephone call voice processing method of the second embodiment have the same effects as those of the first embodiment, and can set a boost amount corresponding to the mask material, thereby making it possible to realize telephone calls that are easier for the listener to hear.

[0076] [Third embodiment] A third embodiment will be described with reference to Figs. 10 to 12. The third embodiment is an embodiment in which a microphone is used as a caller status collection device for collecting caller status information to detect whether the caller is wearing a mask, and the caller status information uses call voice data collected by the microphone (voice uttered by the caller and collected by the microphone). Fig. 10 is a flowchart showing a call voice processing method according to the third embodiment. Each step of the processing flowchart is executed by the processor 30 of the smartphone 1.

[0077] In the flowchart of Fig. 10, the steps performing the same processes as those in the flowchart of Fig. 7 are given the same numbers (S10 to S17). S30 is a voice analysis process consisting of S31 to S37. The voice analysis process S30 may be a multi-process executed in parallel with each step of the call (S15, S16).

[0078] The voice analysis process S30 starts immediately after the caller starts a call (S15) (when the caller is connected to the other party via an electric communication means). The mask detection unit 4220 acquires the voice during the call (S31) and extracts vowels such as "aiueo" (S32). Next, consonants such as "katasafusu" (S33) are extracted. The level ratio between the vowels and consonants is calculated in a high-low level ratio (= low-frequency level / high-frequency level) calculation step (S34), and the class of the high-low level ratio is determined (S35).

[0079] FIG. 11 is an explanatory diagram showing the classification process of the voice analysis process. In the classification process, data of high and low level ratios is classified into several classes. The data in FIG. 11 that associates level ratios with classes is called level ratio class data. In FIG. 11, the level ratio class data is further associated with voice attenuation. Therefore, by finding the level ratio of the speech and determining the class to which it belongs, the voice attenuation associated with that class can be found. Therefore, the step of determining the class of the level ratio corresponds to the voice attenuation estimation step.

[0080] In FIG. 11, the vertical axis is the high-low level ratio, and the horizontal axis is the frequency. The frequency of occurrence of the calculated high-low level ratio data is plotted. From the plotted data group, the boundaries and representative values ​​of classes 0 to 3 are determined. Class 0 is data without a mask with the smallest high-low level ratio, and class 3 is the class with the largest high-low level ratio. The ratio of the representative value of class 0 to the representative values ​​of classes 1 to 3 becomes boost amount 1 to boost amount 3. Classification may be performed based on unsupervised learning processing in which data is clustered by machine learning. At this time, the data is classified into multiple classes according to the material of the mask, etc. The high-low level ratio varies due to the influence of individual differences, the degree of closeness of the mask when worn, etc. Therefore, if the high-low level ratio itself is made to correspond to the boost amount, the boost amount may become unstable, resulting in no improvement in the difficulty of hearing the call. Classification reduces the influence of individual differences, the degree of closeness of the mask when worn, etc.

[0081] Mask detection unit 4220 calculates the boost amount according to the class and obtains a boost control signal. The boost control signal is sent to the boost control step of S16. In addition to immediately reflecting the change in class in the boost control signal, the change may be reflected in consideration of a time transition, thereby reducing frequent changes in the boost amount.

[0082] Mask detection unit 4220 updates class data 434 in data storage area 430 in Fig. 2. Acquisition of voice during a call is performed not only during one call but over multiple calls. In multiple calls, voice in multiple forms, such as with and without masking, is acquired and used as class data. At this time, registered voice data 433 may be referenced to determine whose voice the voice is, and class data 434 may be handled for each individual voice.

[0083] According to this embodiment, when the mask detection unit 4220 uses a voice analysis process to detect the presence or absence of a mask, it is not necessary to capture the face of the caller using the in-camera 72. Therefore, automatic mask detection can be performed even in a situation where, for example, while driving a car, the call device is placed in a storage box, the caller wears a headset with a microphone and earphones, and the call device and headset are paired by close-proximity communication and used.

[0084] FIG. 12 is a flowchart showing a process flow in which image analysis and audio analysis are used in combination in mask detection processing.

[0085] When the call voice processing method is started (S11: YES or S12: YES), an image analysis process (S20) is executed, and during the call after the call starts (S15), a voice analysis process (S30) is executed. As a result, a mask detection process is achieved that satisfies both the immediacy of the image analysis process (S20) and the responsiveness to changes by the voice analysis process (S30).

[0086] As described above, according to the third embodiment, by using the voice analysis process, the mask detection process is possible even in a situation where the in-camera 72 is not effective. It is also possible to deal with changes such as putting on and taking off a mask during a call. Furthermore, by using it in combination with the image analysis process, it is possible to satisfy immediacy and responsiveness to changes.

[0087] [Fourth embodiment] A fourth embodiment will be described with reference to Fig. 13 to Fig. 15. In the fourth embodiment, a head mounted display (also referred to as HMD) is used as a communication device. In the fourth embodiment, a microphone 107 provided in the HMD 100 is used as a caller state collecting device, and a call voice collected by the microphone 107 is used as caller state information.

[0088] 13 is an external view of an HMD 100, which is one form of a communication device. The HMD 100 includes a frame housing including a left temple 130A, a right temple 130B, a front frame 130C, and a nose pad 130D. The front frame 130C includes a camera 101, a distance measurement sensor 102, a left projector 104A, a right projector 104B, and a screen 104C. The left temple 130A includes a left speaker 106A and a microphone 107. The right temple 130B includes a controller 103. A caller wears the HMD 100 on his or her face using the frame housing. Note that the locations of the components may differ from those shown in FIG. 13.

[0089] The left projector 104A, the right projector 104B, and the screen 104C form an image display unit, but instead of these, a semi-transmissive screen or a non-transmissive display may be used. With a semi-transmissive screen, the user visually recognizes the forward background through the semi-transmissive screen, but with a non-transmissive display, the user recognizes the forward background by displaying a camera image capturing the forward background on the non-transmissive display.

[0090] The camera 101 is attached so as to capture an image of the background in front of the user's line of sight, and the distance measurement sensor 102 measures the distance to the background.

[0091] 14 is a hardware configuration diagram of the HMD 100. The HMD 100 includes a camera 101, a distance measurement sensor 102, a left projector 104A, a right projector 104B, a semi-transparent screen 104C, operation keys 105, a left speaker 106A, a right speaker 106B, a microphone 107, a storage 140 built into a controller 103, a GPS receiver 151, a geomagnetic sensor 152, an acceleration sensor 153, a gyro sensor 154, a LAN communicator 161, a mobile communicator 162, and a short-range wireless communicator 163, and the components are connected to each other by a bus 108.

[0092] The storage 140 includes a ROM 141 , a non-volatile memory 142 , and a RAM 143 .

[0093] The controller 103 takes in camera images captured by the camera 101 and distance images measured by the distance measuring sensor 102, and supplies them to a storage 140 and a processor 113 inside the controller 103. The controller 103 also incorporates a GPS receiver 151, a group of sensors such as a gyro sensor, and a communication unit. The controller 103 also creates images to be projected by the left projector 104A and the right projector 104B, and sounds to be output to the left speaker 106A and the right speaker 106B.

[0094] 15 is a block diagram showing programs and data stored in the non-volatile memory 142 of the HMD 100. The non-volatile memory 142 includes a basic operation program 521, an HMD program 522, a call program 523, and a data storage area 524. The processor 113 loads the basic operation program 521, the HMD program 522, and the call program 523 in the RAM 143 and executes them with the processor 113. The data storage area 524 stores data required to execute the basic operation program 521, the HMD program 522, and the call program 523. The call program 523 includes a mask detection unit 4220, a boost control unit 4221, and a call processing unit 4222, as in the first embodiment.

[0095] When the call processing unit 4222 detects a push signal of the call application icon 22 or an incoming call signal, the processor 113 executes the function of the mask detection unit 4220. The mask detection unit 4220 performs voice analysis based on the call voice from the microphone 107, and outputs the mask presence / absence detection result to the boost control unit 4221. The boost control unit 4221 outputs a boost control signal according to the mask presence / absence detection result to the booster 95. The booster 95 performs high-frequency boost processing on the call voice acquired from the microphone 107, and outputs and transmits it from the call processing unit 4222 to the communication device.

[0096] 16 is a diagram illustrating a call scene using the HMD 100. A caller 2 wears the HMD 100 and captures an image of the front background with a camera 101 of the HMD 100. The distance to the front background is measured with a distance measuring sensor 102. These data are sent to an HMD management server 9 connected to a wide area network 7.

[0097] The manager 191 shares these data with the information communication device 192. When the foreground background is, for example, a facility maintenance site, the manager 191 conveys instructions to the caller 2 by voice and image.

[0098] The images are three-dimensional Argument Reality (AR) objects, and are displayed three-dimensionally in a multi-layered manner so as to be in contact with the equipment in the background in front of the HMD 100.

[0099] The caller 2 responds to the instructions of the administrator 191. One form of response is voice. Questions about the instructions are also made by voice. The caller 2's voice is transmitted over a wireless LAN by voice communication using an IP protocol. The voice communication does not require a call procedure such as dialing, and the communication state is always maintained. As long as the communication state is maintained, the call program is executed, and mask detection processing and boost control according to the result are executed. As the mask detection processing, it goes without saying that voice analysis processing can be applied, but image analysis processing may also be applied by adding an in-camera to the HMD 100 that captures the caller's face. Also, the camera of a smartphone may be used by pairing with a device such as a smartphone.

[0100] As described above, according to the fourth embodiment, an HMD can be used as a communication device.

[0101] The present invention is not limited to the embodiments described above in Fig. 1 to Fig. 16, and it is possible to replace a part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention, and furthermore, the numerical values ​​and messages appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0102] Furthermore, communication information terminals are not limited to calls by the terminal's owner alone, but may also be used by acquaintances, etc. In such cases, improved calls can be realized by identifying the caller through face recognition or voice recognition during the call, preparing class data for multiple individuals, and applying optimal voice correction to the caller.

[0103] In the above description, the smartphone 1 and the HMD 100 are taken as examples of communication devices, but devices to which the present invention can be applied are not limited to the smartphone 1 and the HMD 100.

[0104] For example, the telephone call voice processing method may be applied to a Web conference system in which a camera and a microphone are connected to a personal computer. In this case, the telephone call program 422 is installed in the personal computer in advance. Then, the camera captures an image of a caller positioned in front of the microphone, and based on the captured image data, it is detected whether the caller is wearing a mask or not, and the same boost control as described above is executed according to the detection result.

[0105] Also, when participating in a Web conference with the camera turned off, the processor of the personal computer may execute a voice analysis process based on the voice data collected by the microphone to detect whether or not a mask is being worn, and may perform boost control according to the result. In a Web conference, participants may participate with the camera either on or off, depending on their preference, so the call program 422 may execute a mask detection method by switching between image analysis processing and voice analysis processing based on the camera on / off signal.

[0106] In this way, the present invention can be applied to any device that connects at least one of a camera or a microphone, a processor having a calling function, a storage that stores the calling program 422, and a controller including a booster.

[0107] In addition, while the smartphone 1 uses the in-camera 72 as the camera for capturing an image of the caller, in a Web conferencing system the camera for capturing an image of the caller may be a Web camera that is detachably connected to a personal computer or a built-in camera that is integrally formed with a personal computer, and the type of camera for capturing an image of the caller is appropriately selected depending on the device to which the present invention is applied.

[0108] In addition, some or all of the functions of the invention may be implemented in hardware, for example by designing them as an integrated circuit. They may also be implemented in software by a microprocessor unit, CPU, etc. interpreting and executing an operating program. Furthermore, there is no restriction on the scope of software implementation, and hardware and software may be used together. [Explanation of symbols]

[0109] 1, 1A: Smartphone 2, 2A: Caller 3: Mask 4, 4A: Mobile communication base station 5: Mobile communication network 6, 6A: Access point 7: Wide area network 8: Call service server 8a: Mask database 9: HMD management server 20: Home screen 20a: Dial screen 20b:Display screen 21: Icon buttons 22: Call app icon 23: Numeric keypad 24: Start dial button 25: Call permission button 26: Call rejection button 27: Information 30: Processor 31: System bus 40: Storage 41: ROM 42: Non-volatile memory 43: RAM 51: GPS receiver 52: Geomagnetic sensor 53: Acceleration sensor 54: Gyro sensor 60, 60A: Wireless LAN signal 61:LAN communication device 62: Mobile communication device 63: Near field wireless communication device 71: Display 72: In-camera 73: Rear camera 81: Mike 82: Speaker 83: Speaker 91: Touch sensor 92: Operation key 95: Booster 100: HMD 101: Camera 102: Distance sensor 103: Controller 104A: Left projector 104B: Right projector 104C: Screen 105: Operation key 106A: Left speaker 106B: Right speaker 107: Mike 108: Bus 113: Processor 130A: Left vine 130B: Right vine 130C: Front frame 130D: Nose pad 140: Storage 141: ROM 142: Non-volatile memory 143: RAM 151: GPS receiver 152: Geomagnetic sensor 153: Acceleration sensor 154: Gyro sensor 161 :LAN communication device 162: Mobile communication devices 163: Near field wireless communication device 191: Administrator 192: Information and communication devices 421: Basic operation program 422: Call program 430: Data storage area 431: Registered face data 432 : Mask Database 433: Registration voice data 434: Class data 521: Basic operation program 522: HMD Program 523: Call program 524: Data storage area 4220: Mask detector 4221: Boost control unit 4222: Call processing unit

Claims

1. A telephone device using electric communication, The communication device includes a microphone, a communication unit for transmitting and receiving a call voice, a booster for boosting a high-frequency level in a call frequency band, a camera for collecting imaging data to detect whether or not a caller is wearing a mask, and a processor connected to each of the microphone, the communication unit, the booster, and the camera; The processor, a mask detection step of analyzing the imaging data to detect whether the caller is wearing a mask, the step of executing a facial recognition process on the imaging data, recognizing a caller face image in which the face of the caller is captured, and detecting whether the caller is wearing a mask based on whether a mask image in which a mask is captured can be detected from the caller face image; a sound attenuation estimation step of estimating a sound attenuation amount corresponding to the mask image by referring to a mask database in which a type of mask is associated with a sound attenuation amount when the mask is worn; a boost control step of outputting a boost control signal to the booster to control a high-frequency boost of the call voice collected by the microphone according to a detection result of whether the caller is wearing a mask, the boost control step determining a boost amount based on the estimated voice attenuation amount and outputting the boost control signal; a transmitting step of transmitting the call voice after the boost control process by the booster from the communication device; A communication device characterized by executing the above.

2. The communication device according to claim 1, The processor, When the communication device transmits a call request from the communication unit, or when the communication unit receives a call request from another communication device and the communication device starts a response, a mask detection process is started in the mask detection step. A communication device characterized by:

3. A telephone device using electric communications, The communication device includes a microphone, a communication unit for transmitting and receiving a call voice, a booster for boosting a high-frequency level in a call frequency band, a camera for collecting imaging data to detect whether or not a caller is wearing a mask, and a processor connected to each of the microphone, the communication unit, the booster, and the camera; The processor, a mask detection step of analyzing the imaging data to detect whether the caller is wearing a mask, the step of executing a facial recognition process on the imaging data, recognizing a caller face image in which the face of the caller is captured, and detecting whether the caller is wearing a mask based on whether a mask image in which a mask is captured can be detected from the caller face image; a boost control step of outputting a boost control signal to the booster, the boost control signal controlling a high-frequency boost of the call voice collected by the microphone, according to a detection result of whether the caller is wearing a mask; a transmitting step of transmitting the call voice after the boost control process by the booster from the communication device; When the communication device transmits a call request from the communication device, or when the communication device receives a call request from another communication device and the communication device starts a response, the processor starts a mask detection process in the mask detection step, registers the caller's face image at the time the mask detection process is started, generates new imaging data with the camera while the call is ongoing, executes face recognition processing on the new imaging data to recognize a new face image, determines whether the new face image matches the registered caller's face image, and if they match, detects again whether the caller is wearing a mask based on the new face image, and if they do not match, continues to use the detection result of whether the caller is wearing a mask based on the caller's face image. A communication device characterized by:

4. A telephone communication device using electric communications, The communication device includes a microphone, a communication unit that transmits and receives a call voice, a booster that boosts a high-frequency level in a call frequency band, and a processor connected to each of the microphone, the communication unit, and the booster that collects the call voice to detect whether or not the caller is wearing a mask; The processor, a mask detection step of detecting whether the caller is wearing a mask by analyzing the call voice collected by the microphone; a boost control step of outputting a boost control signal to the booster, the boost control signal controlling a high-frequency boost of the call voice collected by the microphone, according to a detection result of whether the caller is wearing a mask; a transmitting step of transmitting the call voice after the boost control process by the booster from the communication device; In the mask detection step, a low-frequency level and a high-frequency level in the call frequency band are detected for the call voice collected by the microphone, a level ratio between the low-frequency level and the high-frequency level is calculated, and whether or not the mask is worn is detected based on the level ratio. A communication device characterized by:

5. The communication device according to claim 4, The processor, In the mask detection step, a class to which the level ratio of the call voice collected by the microphone belongs is determined by referring to level ratio class data in which level ratios are classified in advance, and in the boost control step, a boost amount is determined according to the determined class, and the boost control signal is output. A communication device characterized by:

6. A telephone device using electric communications, The communication device includes a microphone, a communication unit that transmits and receives a call voice, a booster that boosts a high-frequency level in a call frequency band, a caller status collection device that collects caller status information to detect whether the caller is wearing a mask, and a processor connected to each of the microphone, the communication unit, the booster, and the caller status collection device; The processor, a mask detection step of detecting whether the caller is wearing a mask by analyzing the caller status information; a boost control step of outputting a boost control signal to the booster, the boost control signal controlling a high-frequency boost of the call voice collected by the microphone, according to a detection result of whether the caller is wearing a mask; a transmitting step of transmitting the call voice after the boost control process by the booster from the communication device; A camera is used as the caller status collection device, and image data captured by the camera is used as the caller status information. In the mask detection step, the processor executes face recognition processing on the image data to recognize a caller face image in which the face of the caller is captured, and detects whether or not the mask is being worn based on whether or not a mask image in which a mask is captured can be detected from the caller face image. Furthermore, the processor uses the microphone as the caller status collection device and the call voice collected by the microphone as the caller status information, and in the mask detection step, if the camera is activated, executes face recognition processing on the imaging data, recognizes a facial image in which the caller's face is captured, and detects whether or not the mask is being worn based on whether or not a mask image in which a mask is captured can be detected from the facial image, and if the camera is activated, detects a low-frequency level and a high-frequency level of the call voice collected by the microphone, calculates a level ratio between the low-frequency level and the high-frequency level, and detects whether or not the mask is being worn based on the level ratio. At the start of a call, the system uses face recognition via camera photography to determine whether the face is masked or not, and during the call, the system compares the low and high frequency levels of the caller's voice band. A communication device characterized by:

7. The communication device according to claim 1, The communication device transmits and receives call data including the call voice via an IP network. A communication device characterized by:

8. A method of processing voice signals to be executed by a processor in a telephone device using telecommunications, comprising: The communication device includes a microphone, a communication unit for transmitting and receiving a call voice, a booster for boosting a high-frequency level in a call frequency band, a camera for collecting imaging data to detect whether or not a caller is wearing a mask, and a processor connected to each of the microphone, the communication unit, the booster, and the camera; The processor, a mask detection step of analyzing the imaging data to detect whether the caller is wearing a mask, the step of executing a facial recognition process on the imaging data, recognizing a caller face image in which the face of the caller is captured, and detecting whether the caller is wearing a mask based on whether a mask image in which a mask is captured can be detected from the caller face image; a sound attenuation estimation step of estimating a sound attenuation amount corresponding to the mask image by referring to a mask database in which a type of mask is associated with a sound attenuation amount when the mask is worn; a boost control step of outputting a boost control signal to the booster to control a high-frequency boost of the call voice collected by the microphone according to a detection result of whether the caller is wearing a mask, the boost control step determining a boost amount based on the estimated voice attenuation amount and outputting the boost control signal; A transmitting step of transmitting the call voice after the boost control process by the booster from the communication device. A speech processing method comprising:

Images