Communication support device, communication support method, program, and transmitting computer

Abstract

To provide a communication support device that can improve the function of supporting communication between target individuals. [Solution] A communication support device 11 having a processor 33 that performs processing and being communicably connected to a healthy person terminal 12 used by a healthy person A1 and a disabled person terminal 13 used by a disabled person B2, wherein the processor 33 is configured to perform a first process of acquiring transmission information including the voice of healthy person A1 from the healthy person terminal 12, a second process of creating converted information including image information consistent with the intention of healthy person A1 contained in the voice of the transmission information, and a third process of sending the converted information to the disabled person terminal 13.

Description

Technical Field

[0004] , , , , , , , ,

[0005] , , , ,

[0001] The present disclosure relates to a communication support device, a communication support method, a program, and a transmitting computer for supporting communication between target persons.

Background Art

[0002] A system, which is an example of a communication support device for supporting communication between target persons, is described in Patent Document 1. The system described in Patent Document 1 aims to eliminate the need for a person with hearing impairment to rely on sign language or written communication for smooth communication.

[0003] The system described in Patent Document 1 includes a speech recognition means for acquiring speech information, which is the voice of a speaker, by speech recognition and converting the speech information into text data, a learning means for integrating vibration data of a bone conduction microphone and video data from a built-in camera of a smartphone and learning the correlation between the mouth movement and the speech of the speaker, and a speaker for flowing a synthetic speech corresponding to the text data recognized by the speech recognition means in cooperation with the smartphone, and is a system for assisting the communication of a person with hearing impairment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The inventor of the present application recognized the problem that the system described in Patent Document 1 still has an insufficient function of supporting communication between target persons because the speech information of the speaker is only converted into text data and the synthetic speech corresponding to the text data flows from the speaker.

[0006] The purpose of this disclosure is to provide a communication support device, a communication support method, a program, and a transmitting computer that can improve the function of supporting communication between subjects. [Means for solving the problem]

[0007] This embodiment discloses a communication support device having a processor that performs processing and being communicably connected to a first target terminal used by a first target and a second target terminal used by a second target, wherein the processor is configured to perform a first process of acquiring transmission information including the voice of the first target from the first target terminal, a second process of creating converted information including image information consistent with the intent of the first target contained in the voice of the transmission information, and a third process of sending the converted information to the second target terminal. [Effects of the Invention]

[0008] The communication support device of this embodiment has improved functionality in supporting communication between the target individuals. [Brief explanation of the drawing]

[0009] [Figure 1] This is a conceptual diagram showing an example configuration of a communication support system. [Figure 2] This is a conceptual diagram of a communication support device that constitutes part of a communication support system. [Figure 3] This is a conceptual diagram illustrating a specific example of a communication support device. [Figure 4] This flowchart shows an example of a process performed by the communication support system. [Figure 5] This flowchart shows an example of processing performed by the communication support system, specifically example 2. [Modes for carrying out the invention]

[0010] (overview) In this embodiment, several specific examples of a communication support system, a communication support device, a communication support method, and a program are described with reference to Figures 1 and 2. The communication support system 10 may be implemented by a communication support device 11, a healthy person's terminal 12, a disabled person's terminal 13, an external computer 14, etc. The support device 11 may be connected to the healthy person's terminal 12, the disabled person's terminal 13, and the external computer 14 via a network 15. In this embodiment, the communication support device 11 is abbreviated as "support device 11" for convenience.

[0011] The healthy user terminal 12 is a computer used and operated by a healthy user A1, who is an example of a target user. Healthy user A1 may be a person without disabilities such as hearing impairment, developmental disability, speech impairment, or visual impairment. Healthy user A1 may also be a person who can communicate smoothly in everyday language, for example, a person between the ages of 10 and 79.

[0012] The disabled person terminal 13 is a computer used and operated by disabled person B1, who is an example of a target individual. Disabled person B1 may have one or more disabilities from among hearing impairment, developmental disability, speech impairment, visual impairment, intellectual disability, etc. Also, disabled person B1 may be a person who has difficulty with daily verbal communication, for example, a person under 9 years old or a person over 80 years old.

[0013] Network 15 is a communication line for transmitting various types of information, and Network 15 may be implemented by one or more communication lines from, for example, an internetwork, a LAN (Local Area Network), a CAN (Campus Area Network), a WAN (Wide Area Network), etc. Various types of information may include image information, sound information, text data, commands, requests, folders, files, etc. Image information may include video (images), still images (images), photographs, storyboards (illustrations), etc. Sound information may include spoken voice (human voice), artificial voice, music, etc. Text data may include sentences, characters, symbols, etc.

[0014] Network 15 may consist of base stations, radio towers, communication antennas, repeaters, routers, modems, communication circuits, communication cables (copper wires, optical fibers), etc. Network 15 may be implemented by one or more communication means, including wireless communication and wired communication. Wireless communication includes short-range wireless communication. Short-range wireless communication includes, for example, wireless LAN (Wi-Fi®) and Bluetooth®. Wireless communication includes electrical signals, optical signals, radio waves, infrared rays, satellite communication, etc. Signals used for communication include digital signals and analog signals.

[0015] The communication support system 10 of this embodiment helps healthy person A1 and disabled person B1 communicate smoothly in a predetermined environment by reducing communication barriers between them. The predetermined environment may include everyday living spaces, workplaces, public spaces, etc.

[0016] Furthermore, the able-bodied terminal 12 and the disabled terminal 13 may communicate with each other via the support device 11. Also, if the able-bodied terminal 12 and the disabled terminal 13 cannot communicate with each other via the support device 11 due to various communication failures, the able-bodied terminal 12 and the disabled terminal 13 may communicate with each other via short-range wireless communication without going through the support device 11.

[0017] (Configuration of a healthy person's terminal) The healthy person's terminal 12 includes, for example, a portable computer and a fixed computer. The portable computer includes a smartphone, a tablet computer, AR (Augmented Reality) glasses, and the like. The fixed computer includes a desktop computer, a notebook computer, and the like.

[0018] As shown in FIG. 1, the healthy person's terminal 12 includes a main body (casing), a ROM 16, a processor 17, a main storage device 18, an auxiliary storage device 19, an input device 20, an output device 21, a communication device 22, and the like. The ROM 16 is a read-only memory for data, and once data is written during manufacturing, it will not be changed. The ROM 16 is a non-volatile storage device.

[0019] The processor 17 may be provided inside the main body and may be composed of a central processing unit (CPU (Central Processing Unit)) in which an arithmetic unit (arithmetic circuit) and a control unit (control circuit) are integrated. Also, the processor 17 may be a multiprocessor. The processor 17 may be, for example, a combination of two or more elements among a CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), and a GPU (Graphics Processing Unit).

[0020] Furthermore, the processor 17 is communicably connected to the ROM 16, the main storage device 18, the auxiliary storage device 19, the input device 20, the output device 21, and the communication device 22 via a communication bus 23. The processor 17 has a configuration that controls including other devices and circuits provided inside the main body and devices and circuits provided outside the main body.

[0021] The processor 17 processes and executes various types of information by running non-temporary programs. These types of information may include information, text data, sound information, image information, commands, requests, signals, etc. The processing performed by the processor 17 includes calculations, decisions, comparisons, control, etc.

[0022] The processes executed by the processor 17 include processing various types of information input by operating the input device 20, controlling the output device 21, reading programs and various types of information from the auxiliary storage device 19, processing various types of information obtained from the support device 11, and transmitting various types of information to the support device 11. The processor 17 may also perform processes such as adding new types of information to the auxiliary storage device 19, overwriting and updating existing information, and creating new information by combining multiple extracted pieces of information.

[0023] The main memory 18 is a volatile memory and may be implemented by semiconductor memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The processor 17 can perform processes to write various types of information to the main memory 18 and processes to read various types of information from the main memory 18. The main memory 18 functions as a work area and a buffer area for storing programs, data, instructions, etc., retrieved from the auxiliary memory 19 when the processor 17 executes processing of programs, data, instructions, etc.

[0024] The auxiliary storage device 19 is a non-temporary storage medium. The auxiliary storage device 19 may be implemented as, for example, a magnetic disk, an optical disk, or flash memory. Non-temporary applications and programs are stored in the auxiliary storage device 19. In addition, various types of information processed by the processor 17 may be stored in the auxiliary storage device 19. The various types of information stored in the auxiliary storage device 19 may include the processing results of the processor 17, various data, various types of information and commands input from the input device 20, control contents to the output device 21, etc.

[0025] The processor 17 performs various processes by reading and executing non-temporary programs stored in the auxiliary storage device 19. For example, the processor 17 processes various types of information input through the input device 20 and various types of information input via the communication device 22. The processor 17 also has the function of controlling the output device 21. Furthermore, the processor 17 stores the processing results in the auxiliary storage device 19.

[0026] The input device 20 is a device operated by a healthy person A1, and may consist of a keyboard 20A, a mouse 20B, a microphone 20C, a camera 20D, a reader 20F, a display 20E, etc. Healthy person A1 can input various types of information to the healthy person terminal 12 by operating the input device 20. Healthy person A1 can input sound information using the microphone 20C.

[0027] The microphone 20C is an electronic component that converts acquired sound information into an electrical signal and outputs it. The microphone 20C is connected to the communication bus 23 by either a wireless or wired communication system. The camera 20D is connected to the communication bus 23 by either a wireless or wired communication system. The camera 20D captures an object and generates image information. The image information includes video, still images, photographs, etc. The image information generated by the camera 20D is processed by the processor 17 and stored in the auxiliary storage device 19.

[0028] The reading device 20F is a device that reads various types of image information and converts it into electrical signals. The reading device 20F may be fixed, manual, optical, or any other type.

[0029] Furthermore, the display 20E can be implemented using, for example, a liquid crystal display or an organic electroluminescent display. The user screen of the display 20E will display text data, image information, operation buttons, operation tabs, etc. The display 20E can also be considered a monitor.

[0030] When the input device 20 is used, the healthy user terminal 12 may perform the following actions: input various types of information into the healthy user terminal 12, execute various processes on the healthy user terminal 12, send various types of information from the healthy user terminal 12 to the support device 11 and the disabled user terminal 13, the healthy user terminal 12 acquires various types of information from the support device 11 and the disabled user terminal 13, output various types of information from the output device 21, etc. The various types of information may include image information, sound information, data, commands, signals, etc.

[0031] The output device 21 may include devices such as a display 20E, an audio information output device 21A, and a printer 21B. The display 20E is the same as the display 20E of the input device 20. The audio information output device 21A is a device that converts electrical signals into audio information and outputs it. The audio information output device 21A is implemented by a speaker, headphones, earphones, etc. The audio information output device 21A is connected to the communication bus 23 by wireless or wired communication. The printer 21B prints information and data onto paper using toner or ink and outputs it.

[0032] The communication device 22 includes devices, equipment, and standards for connecting the healthy person terminal 12 to the network 15 via at least one of either a wireless communication system or a wired communication system. The devices and equipment for connecting the healthy person terminal 12 to the network 15 may include cables, antennas, routers, modems, repeaters, communication circuits, communication ports, communication connectors, communication hubs, etc.

[0033] (Configuration of devices for people with disabilities) The disabled person's terminal 13 includes, for example, a portable computer and a fixed computer. Portable computers include smartphones and tablet computers. Fixed computers include desktop computers, notebook computers, etc.

[0034] As shown in Figure 1, the disabled terminal 13 includes a main unit (casing), ROM 24, processor 25, main memory 26, auxiliary memory 27, input device 28, output device 29, communication device 30, etc. The ROM 24 is a data read-only memory, and the data written to it during manufacturing is not modified afterward. The ROM 24 is a non-volatile memory device.

[0035] The processor 25 may be located inside the main unit and consist of a central processing unit (CPU) that integrates an arithmetic unit (arithmetic circuit) and a control unit (control circuit). The processor 25 may also be a multiprocessor. For example, the processor 25 may be a combination of two or more elements from among a CPU, MPU (Micro Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), and GPU (Graphics Processing Unit).

[0036] Furthermore, the processor 25 is communicatively connected to the ROM 24, main memory 26, auxiliary memory 27, input device 28, output device 29, and communication device 30 via the communication bus 31. The processor 25 has a configuration that comprehensively controls other devices and circuits provided inside the main unit, as well as devices and circuits provided outside the main unit.

[0037] The processor 25 processes and executes various types of information by running non-temporary programs. These types of information may include information, text data, sound information, image information, commands, requests, signals, etc. The processing performed by the processor 25 includes calculations, judgments, comparisons, and control.

[0038] The processes executed by the processor 25 include processing various types of information input by operating the input device 28, controlling the output device 29, reading programs and various types of information from the auxiliary storage device 27, processing various types of information obtained from the support device 11, and transmitting various types of information to the support device 11. The processor 25 may also perform processes such as adding new types of information to the auxiliary storage device 27, overwriting and updating existing information, and creating new information by combining multiple extracted pieces of information.

[0039] The main memory 26 is a volatile memory and may be implemented by semiconductor memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The processor 25 can perform processes to write various types of information to the main memory 26 and processes to read various types of information from the main memory 26. The main memory 26 functions as a work area and buffer area for storing programs, data, instructions, etc., retrieved from the auxiliary memory 27 when the processor 25 executes processing of programs, data, instructions, etc.

[0040] The auxiliary storage device 27 is a non-temporary storage medium. The auxiliary storage device 27 may be implemented as, for example, a magnetic disk, an optical disk, or flash memory. Non-temporary applications and programs are stored in the auxiliary storage device 27. In addition, various types of information processed by the processor 25 may be stored in the auxiliary storage device 27. The various types of information stored in the auxiliary storage device 27 may include the processing results of the processor 25, various data, various types of information and commands input from the input device 28, control contents to the output device 29, etc.

[0041] The processor 25 performs various processes by reading and executing non-temporary programs stored in the auxiliary storage device 27. For example, the processor 25 processes various types of information input from the input device 28 and various types of information input via the communication device 30. The processor 25 also has the function of controlling the output device 29. Furthermore, the processor 25 stores the processing results in the auxiliary storage device 27.

[0042] The input device 28 is a device operated by the disabled person B1, and may consist of a keyboard 28A, a mouse 28B, a microphone 28C, a camera 28D, a reader 28F, a display 28E, etc. By operating the input device 28, the disabled person B1 can input various types of information into the disabled person terminal 13. The disabled person B1 can input sound information using the microphone 28C.

[0043] The microphone 28C is an electronic component that converts acquired sound information into an electrical signal and outputs it. The microphone 28C is connected to the communication bus 31 by either a wireless or wired communication system. The camera 28D is connected to the communication bus 31 by either a wireless or wired communication system. The camera 28D captures images of objects and generates image information. The image information includes videos, still images, photographs, etc. The image information generated by the camera 28D is processed by the processor 25 and stored in the auxiliary storage device 27.

[0044] The reading device 28F is a device that reads various types of image information and converts it into electrical signals. The reading device 28F may be fixed, manual, optical, or any other type.

[0045] Furthermore, the display 28E can be implemented using, for example, a liquid crystal display or an organic electroluminescent display. The user screen of the display 28E displays text data, image information, operation buttons, operation tabs, etc. The display 28E can also be considered a monitor.

[0046] When the input device 28 is used, the disabled terminal 13 may perform the following actions: input various types of information into the disabled terminal 13, execute various processes on the disabled terminal 13, send various types of information from the disabled terminal 13 to the support device 11 and the able-bodied terminal 12, the disabled terminal 13 acquires various types of information from the support device 11 and the able-bodied terminal 12, output various types of information from the output device 29, etc. The various types of information may include image information, sound information, data, commands, signals, etc.

[0047] The output device 29 may include devices such as a display 29F, an audio information output device 29A, and a printer 29B. The display 29F is the same as the display 29F of the input device 28. The audio information output device 29A is a device that converts electrical signals into audio information and outputs it. The audio information output device 29A is implemented by a speaker, headphones, earphones, etc. The audio information output device 29A is connected to the communication bus 31 by wireless or wired communication. The printer 29B prints information and data onto paper using toner or ink and outputs it.

[0048] The communication device 30 includes devices, equipment, and standards for connecting the disabled terminal 13 to the network 15 via at least one of either a wireless communication system or a wired communication system. The devices and equipment for connecting the disabled terminal 13 to the network 15 may include cables, antennas, routers, modems, repeaters, communication circuits, communication ports, communication connectors, communication hubs, etc.

[0049] (Example of support device configuration) The support device 11 is located at a predetermined IP address on the network 15. The support device 11 is a computer implemented by various hardware and various software. The software includes an operating system and applications. The support device 11 may have hardware such as a main unit (casing), ROM (Read Only Memory) 32, processor 33, main memory 34, auxiliary storage 35, media drive device 36, and communication device 37. The ROM 32 is a data read-only memory and the data written to it during manufacturing is not modified. The ROM 32 stores the program that is executed first when the support device 11 starts up, such as the IPL (Initial Program Loader). The ROM 32 is a non-volatile storage device.

[0050] The processor 33 is located in the main body of the support device 11 and may be composed of a central processing unit (CPU) that integrates an arithmetic unit (arithmetic circuit) and a control unit (control circuit). The processor 33 is communicatively connected to the ROM 32, main memory 34, auxiliary memory 35, media drive device 36, and communication device 37 via a communication bus 38. The processor 33 comprehensively controls other devices and circuits located inside the main body, as well as devices and circuits located outside the main body.

[0051] Furthermore, the processor 33 may be a multiprocessor. The processor 33 may be a combination of two or more elements from, for example, a CPU, an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), and a GPU (Graphics Processing Unit).

[0052] The processor 33 may perform various processes by running a program stored in the auxiliary storage device 35. The processes performed by the processor 33 include arithmetic, judgment, learning, comparison, identification, classification, analysis, reasoning, regression, proposal, control, calculation, generation, etc. The processes performed by the processor 33 may also include processing various information, storing various information in the auxiliary storage device 35, reading various information from the auxiliary storage device 35, acquiring various information from the healthy person terminal 12, transmitting various information to the healthy person terminal 12, acquiring various information from the disabled person terminal 13, transmitting various information to the disabled person terminal 13, and acquiring various information from the external computer 14.

[0053] The various types of information processed by the processor 33 may include image information, sound information, text data, commands, requests, signals, files, etc. The processor 33 may also perform processes such as adding new information to the auxiliary storage device 35, overwriting and updating existing information, and combining multiple extracted pieces of information to create new information.

[0054] The main memory 34 is a volatile memory and may be implemented by semiconductor memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The processor 33 can perform processes to write various types of information to the main memory 34 and processes to read various types of information from the main memory 34. The main memory 34 functions as a work area and a buffer area for storing programs, data, instructions, etc., retrieved from the auxiliary memory 35 when the processor 33 processes and executes such programs, data, instructions, etc.

[0055] The auxiliary storage device 35 is a non-temporary storage medium. The auxiliary storage device 35 may also be understood as storage. The auxiliary storage device 35 may be located inside the main unit, or it may be located outside the main unit and connected to the communication bus 38 by wireless or wired communication. The auxiliary storage device 35 operates according to input and output commands from the processor 33.

[0056] The auxiliary storage device 35 may be implemented by, for example, a magnetic disk, an optical disk, or flash memory. A magnetic disk is a storage device that can magnetically write and read various types of data to and from the storage medium 35A, and the storage medium 35A may be implemented by, for example, an HDD (Hard Disk Drive) or a floppy disk. An optical disk is a storage device that can write various types of information to and read various types of information by irradiating the storage medium 35A with laser light. The storage medium 35A may be implemented by, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), or a BD (Blu-ray® Disc). Flash memory is a storage device that uses semiconductor memory, and the storage medium 35A may consist of an SSD (Solid State Drive), a USB memory (USB Flash Drive), or an SD card (Secure Digital Card).

[0057] The auxiliary storage device 35 may have non-temporary programs pre-stored in it. It may also store non-temporary programs read by the media drive device 36. Furthermore, it may install non-temporary programs downloaded from an external computer 14 via the network 15. Additionally, it may store various types of information obtained from the external computer 14.

[0058] The media drive device 36 is hardware into which a non-temporary storage medium 36A is inserted and removed. The storage medium 36A is portable and can be implemented as, for example, a CD-ROM (Compact Disc Read Only Memory), a DVD, etc. The storage medium 36A may store a program that is operated by the processor 33. After the program stored in the storage medium 36A is installed into the auxiliary storage device 35, it is stored in the main memory 34, making the processor 33 able to execute the program. In addition, the storage medium 36A of the media drive device 36 may store various kinds of information. The various kinds of information stored in the storage medium 36A may be stored in the auxiliary storage device 35.

[0059] The processor 33 may implement various functional components shown in Figure 3 by running non-temporary programs and cooperating with various hardware components that make up the support device 11. For example, the processor 33 may implement a website management unit 39, a healthy person information processing unit 40, a disabled person information processing unit 41, a conversion information creation unit 42, various information processing units 43, and an artificial intelligence unit 44.

[0060] The website management unit 39 may provide a website at a predetermined URL (Uniform Resource Locator) on the network 15. The website management unit 39 may also provide communication support services to able-bodied terminals 12 or disabled terminals 13 that access the support device 11 via the website.

[0061] The healthy person information processing unit 40 may process various information acquired from the healthy person terminal 12 and store the processing results in the auxiliary storage device 35. The disabled person information processing unit 41 may process various information acquired from the disabled person terminal 13 and store the processing results in the auxiliary storage device 35.

[0062] The conversion information creation unit 42 may create first conversion information and second conversion information in cooperation with various information processing units 43, the artificial intelligence unit 44, and the auxiliary storage device 35. The created first conversion information and second conversion information may be stored in the auxiliary storage device 35. Furthermore, if there is a request to modify the first conversion information and second conversion information, the conversion information creation unit 42 may modify the first conversion information and second conversion information. The meanings of first conversion information and second conversion information, and the meaning of a request to modify first conversion information and second conversion information will be described later.

[0063] The various information processing units 43 may process various types of information acquired from the external computer 14 and store the processing results in the auxiliary storage device 35. In addition, the various information processing units 43 may process information that is not processed by the website management unit 39, the healthy person information processing unit 40, the disabled person information processing unit 41, the conversion information creation unit 42, and the artificial intelligence unit 44, and store the processing results in the auxiliary storage device 35.

[0064] Furthermore, the various information processing units 43 may, for example, determine whether a person with a disability, B1, who is using a terminal 13 connected to the support device 11, is hearing impaired. Whether or not person with a disability, B1, is hearing impaired can be determined from the user profile of the person with a disability. In addition, the various information processing units 43 may determine whether the first conversion information was approved on the non-disabled person's terminal 12, and whether the second conversion information was approved on the disabled person's terminal 13. Furthermore, the various information processing units 43 may process information about communication (dialogue) that took place between a non-disabled person, A1, and a person with a disability, B1.

[0065] When the program is run, the artificial intelligence unit 44 works in cooperation with the auxiliary storage device 35 to perform various processes. The processes performed by the artificial intelligence unit 44 include learning processes (learning phase) and inference processes (inference phase). The various types of information used by the artificial intelligence unit 44 to perform the processes include image information, sound information, text data, requests, commands, etc.

[0066] The artificial intelligence unit 44 outputs processing results (output data) by processing the input data to be studied using a learning model during learning processing, such as machine learning processing. The artificial intelligence unit 44 also outputs inference results by processing the data to be inferred using a learning model during inference processing.

[0067] The artificial intelligence unit 44 may extract features from the input data using machine learning. Feature extraction from the input data includes classification and regression. Classification involves classifying and identifying features. Features may include regularities in the input data. Regression includes understanding the overall trend of the classified features, predicting the future state of the features, etc. The learning model is stored in the auxiliary memory device 35, and the learning model is trained as the artificial intelligence unit 44 repeats machine learning.

[0068] The machine learning performed by the artificial intelligence unit 44 may include three types: supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the artificial intelligence unit 44 uses a loss function to calculate the error between the processing result and the ground truth data, and trains the learning model based on that error. Specifically, it adjusts weights, biases, etc., to minimize the error. The artificial intelligence unit 44 trains the learning model by comparing the output data with the ground truth data (ground truth labels). The ground truth data used by the artificial intelligence unit 44 may be assigned to classify, analyze, and identify the input data.

[0069] Unsupervised learning is a method of training a model using training data that does not contain correct answers, and it involves classifying the training data into groups of data with similar features. Reinforcement learning involves repeatedly processing input data with a learning model to obtain output data, and training the learning model so that it can maximize rewards by acting according to the output data.

[0070] The artificial intelligence unit 44 can use a neural network as a learning model to realize machine learning. A neural network has an input layer, an intermediate layer (hidden layer), and an output layer. The intermediate layer is configured to make the input information useful for the output layer. Furthermore, the artificial intelligence unit 44 can perform deep learning in realizing the intermediate layer of the neural network. In deep learning, the intermediate layer is multi-layered with multiple types of stages (nodes), and weights and biases are assigned to the functions (features) contained in the information. Deep learning can reduce noise in the information and data input to the learning model because it automatically instructs the intermediate layer to focus on the functions (features).

[0071] Furthermore, the artificial intelligence unit 44 may perform tasks such as analyzing and interpreting text data, generating prompts, and generating sentences of inference results corresponding to the prompts, using the functions of a large-scale language model. The analysis and interpretation of chat content includes analyzing and interpreting text data, and converting audio data into text data for analysis and interpretation.

[0072] Large language models (LLMs) are a type of generative artificial intelligence specifically designed for natural language processing (NLP). LLMs are language models that achieve advanced natural language generation (LNG) by being constructed through deep learning of vast amounts of text data.

[0073] Natural language processing combines processes such as morphological analysis, syntactic analysis, semantic analysis, contextual analysis, and intent analysis to process natural language, enabling machine translation, text summarization, automated speech recognition (ASR), and text-to-speech (TTS). Text-to-speech is a technology that converts text data into speech.

[0074] A language model is a system that predicts the word that follows a given word by understanding the patterns of natural language used by humans and analyzing the probability of word occurrence. Examples of large-scale language models include ChatGPT (Chat Generative Pre-trained Transformer) (registered trademark), PaLM2 (registered trademark), and LaMDA2 (registered trademark). The words and sentences used as instructions to input into these large-scale language models are called prompts.

[0075] When the artificial intelligence unit 44 functions as a generative artificial intelligence unit, it may perform processing in response to input prompts and output various inference results, such as image information, sound information, text data, etc.

[0076] Image information generation is a function that automatically generates original image information that closely resembles the image of the prompt, using various information stored in the auxiliary storage device 35 in response to the input prompt (text format). Examples of artificial intelligence that generate image information include stable diffusion, mid-journey, and Dali-2.

[0077] Video generation is a function that automatically generates an original video that closely resembles the image of a prompt (in text format) by using the information and data stored in the auxiliary storage device 35 in response to an input prompt. Gentoo is a well-known example of artificial intelligence that generates videos. Voice generation is a function that automatically generates original voice data (optimal actions, responses, music, etc.) by using the information and data stored in the auxiliary storage device 35 in response to an input prompt (in voice or text format).

[0078] The artificial intelligence unit 44 of this embodiment may perform preprocessing on the input data during the inference stage. Specific examples of preprocessing will be described later. The artificial intelligence unit 44 of this embodiment may also create first and second transformation information during the inference stage. Furthermore, the artificial intelligence unit 44 of this embodiment may perform processing to modify the first transformation information and processing to modify the second transformation information during the inference stage.

[0079] The communication device 37 includes devices, equipment, and standards for connecting the support device 11 to the able-bodied terminal 12 and the disabled terminal 13 via the network 15 using at least one of either a wireless communication system or a wired communication system. The hardware constituting the communication device 37 may consist of, for example, a network adapter, a network interface card, a communication cable, a communication antenna, a communication port, etc.

[0080] Furthermore, as shown in Figure 3, the auxiliary storage device 35 is implemented by including a website information storage unit 45, a healthy person information storage unit 46, a disabled person information storage unit 47, a conversion information storage unit 48, various information storage units 49, a program storage unit 50, and a model storage unit 51.

[0081] The system is implemented by including a website information storage unit 45, a healthy person information storage unit 46, a disabled person information storage unit 47, a conversion information storage unit 48, various information storage units 49, a program storage unit 50, and a model storage unit 51. The website information storage unit 45 may store hypertext data for the website management unit 39 to configure the website, user screens provided to computers connected to the website, etc.

[0082] The healthy person information storage unit 46 may store various information obtained from the healthy person terminal 12, and the results of processing the various information obtained from the healthy person terminal 12. The disabled person information storage unit 47 may store various information obtained from the disabled person terminal 13, and the results of processing the various information obtained from the disabled person terminal 13.

[0083] The conversion information storage unit 48 may store the first conversion information and second conversion information created by the conversion information creation unit 42, the modified first conversion information and second conversion information, etc. The various information storage unit 49 may store various information used to execute processing in the conversion information creation unit 42, various information processing units 43, and artificial intelligence unit 44, as well as information resulting from processing performed by the various information processing units 43.

[0084] For example, the various information storage units 49 may store various information used by the artificial intelligence unit 44 to determine whether the first transmitted information is clear, and various information used by the artificial intelligence unit 44 to determine whether the second transmitted information is clear. The various information storage units 49 may also store image information, sound information, various texts, etc., corresponding to a predetermined environment. The image information corresponding to a predetermined environment may include storyboards, illustrations, videos, still images, etc., including items, tools, tasks, etc., that are handled at a predetermined frequency or higher in the predetermined environment. The sound information and various text data corresponding to a predetermined environment may include slang, idioms, sentences, phrases, example sentences, requests, demands, answers, replies, etc., that are used at a predetermined frequency or higher in the predetermined environment.

[0085] Furthermore, the various information storage units 49 may store items such as the first conversion information transmitted to the healthy person terminal 12, the result of determining whether or not the first conversion information was approved, and requests for correction of the first conversion information that was not approved, all of which may be linked together.

[0086] Furthermore, the various information storage units 49 may store items such as the second conversion information transmitted to the disabled terminal 13, the result of the judgment as to whether or not the second conversion information was approved, and requests for correction of the second conversion information that was not approved, all of which may be linked together. In addition, the various information storage units 49 may store a history of communication (dialogue) information that took place between the able-bodied person A1 and the disabled person B1.

[0087] The program storage unit 50 may store applications run by the processor 33. Applications stored in the program storage unit 50 include non-temporary programs, manuals, configuration files, data storage files, data, various libraries, etc. The programs stored in the program storage unit 50 may include, for example, programs that implement communication support methods. The model storage unit 51 stores learning models and large-scale language models, etc.

[0088] (Example of processing performed by the communication support system 1) Figure 4 shows an example of processing 1 performed by the communication support system 10. Processing example 1 is an example in which a speech statement, including a work-related request, made by a healthy person A1 in a predetermined environment such as a workplace, for example, a supermarket, is transmitted to a person with a disability B1 via the support device 11.

[0089] In step S30, the support device 11 may provide a website at a predetermined URL on the network 15. Here, healthy person A1 may have the healthy person terminal 12 access the website provided by the support device 11 on the network 15, and have the user screen provided by the support device 11 displayed on the display 20E. Healthy person A1 may also input a user profile into the healthy person terminal 12.

[0090] The user profile includes items such as the name, gender, date of birth, address, email address, occupation and work, and healthy person account of healthy person A1. The healthy person account is information that identifies healthy person A1 and consists of an ID and password. When the healthy person terminal 12 is connected to the support device 11, the healthy person account is transmitted to the support device 11. Therefore, the support device 11 can identify the healthy person terminal 12 and healthy person A1.

[0091] When the support device 11 obtains a user profile from the healthy person terminal 12, it may register the healthy person A1 as a user and store the user profile in the auxiliary storage device 35.

[0092] Alternatively, person B1 with a disability may have the assistive device 11 access a website provided on the network 15 using the disabled person's terminal 13, and display the user interface provided by the assistive device 11 on the display 28E. Person B1 with a disability may also input a user profile into the disabled person's terminal 13.

[0093] The user profile includes items such as the name, gender, date of birth, address, email address, occupation and work, disability account, presence or absence of disability, type and nature of disability, etc. The disability account is information that identifies disability B1 and consists of an ID and password. When the disability terminal 13 is connected to the support device 11, the disability account is transmitted to the support device 11. Therefore, the support device 11 can identify the disability terminal 13 and disability B1. When the support device 11 obtains the user profile from the disability terminal 13, it may register disability B1 as a user and store the user profile in the auxiliary storage device 35.

[0094] Then, in step S10, healthy person A1 may input first communication information into healthy person terminal 12. The first communication information is information that includes the intention that healthy person A1 intends to convey to disabled person B1. The first communication information may include, for example, sound information including speech made by healthy person A1, image information and text data used by healthy person A1 when speaking. The content of the speech may include greetings, requests, inquiries, confirmations, replies and answers, etc.

[0095] A healthy person A1 may input the following as a request to the disabled person B1, for example, audio information and text data such as "Please put down a banana," and image information including a picture of a banana, as the first communication information, into the healthy person terminal 12. Furthermore, in step S10, the healthy person A1 may transmit the first communication information from the healthy person terminal 12 to the support device 11.

[0096] In step S31, the support device 11 may acquire the first transmission information from the healthy person terminal 12 and store the first transmission information in the auxiliary storage device 35.

[0097] Furthermore, the support device 11 may preprocess the first transmitted information in step S31. The preprocessing performed by the support device 11 is a function for estimating the intention of the healthy person A1 contained in the first transmitted information. Specifically, the artificial intelligence unit 44 of the support device 11 may perform preprocessing in cooperation with the auxiliary storage device 35. The preprocessing performed by the artificial intelligence unit 44 may include processing to classify and analyze the meaning and content of sound information, image information, and text data, and processing to extract and analyze unclear information from sound information and text data. The unclear information may include information with unclear (ambiguous) meaning, and sound information that is difficult for healthy person A1 and disabled person B1 to hear (understand).

[0098] Information with unclear meaning includes terms with unclear meaning, such as "that," "this," "it," "the one from earlier," "the big one," "the white one," etc. Information with unclear meaning also includes "information that is missing one or more of the 5W2H (Who, What, When, Where, Why, How)." Sound information that is difficult to hear includes sound information containing noise, sound information with a volume below a predetermined value, sound information containing a language not used in the listener's residential area, etc. Thus, the artificial intelligence unit 44 has the function of determining whether the first transmitted information is clear or unclear by working in cooperation with the auxiliary storage device 35. The artificial intelligence unit 44 may also store the results of the preprocessing in the auxiliary storage device 35.

[0099] The artificial intelligence unit 44 of the support device 11 may create first converted information in step S32 based on the results of preprocessing the first transmitted information. The process of creating the first converted information may include the process of generating image information that is consistent with the intention of the healthy person A1 contained in the sound information of the first transmitted information. Furthermore, the process of creating the first converted information may include the process of converting information that was judged to be unclear in meaning during preprocessing of the sound information and text data contained in the first transmitted information into sound information and text data with clear meaning.

[0100] The process of converting information deemed to have unclear meaning into audio information with clear meaning may include, for example, the process of converting information with unclear meaning into image information with clear meaning or into text data with clear meaning. The process of converting information with unclear meaning into image information with clear meaning may include, for example, the process of representing proper nouns, procedures, etc., contained in audio information with image information, the process of generating multiple images of work procedures contained in audio information, or the process of generating multiple images of story stages contained in audio information.

[0101] The process of converting information with unclear meaning into meaningful text data may include, for example, specifying modifiers, using commas, dividing long sentences into multiple sentences, and changing the order of modifiers when a sentence contains multiple modifiers.

[0102] Furthermore, the process of creating the first conversion information may include a process of amplifying the volume of sound information that is difficult to hear in the sound information included in the first transmission information to a volume exceeding a predetermined value. In addition, the support device 11 may determine in step S32 whether or not the disabled person B1 is hearing impaired. If the disabled person B1 is hearing impaired, the support device 11 may set the volume of the sound information included in the first conversion information to be higher than the volume of the sound information included in the first conversion information if the disabled person B1 is hearing impaired.

[0103] Furthermore, the process for creating the first conversion information may include a process for converting a language not used in the listener's residential area into a language used in the listener's residential area. In addition, the process for creating the first conversion information may include a process for removing noise contained in the sound information. Furthermore, the process for creating the first conversion information may include processes such as candidate generation, candidate evaluation, selection guidance, response candidate sorting, presentation optimization, and selection response processing.

[0104] The candidate generation process involves generating multiple types of response candidates by performing processes such as speech recognition, ambiguity resolution, and information analysis. Information analysis may include analyzing the 5W2H elements contained in the speech information, analyzing the relationships between words contained in the speech information, and analyzing actions and events contained in the speech information. The 5W2H elements are Where, Who, What, Why, How, and How Much.

[0105] The generation of multiple response candidates is achieved by combining various information obtained from healthy terminals 12 and disabled terminals 13, various information obtained from an external computer 14, various information stored in the auxiliary storage device 35, and by modifying or correcting the various information stored in the auxiliary storage device 35.

[0106] The candidate evaluation process may include evaluating multiple types of response candidates based on user profiles, dictionaries, past history, the degree to which the subject understands multiple types of response candidates, the consistency between the input communication information and the response candidates, and the consistency with the individuality of neurotypical individuals and individuals with developmental disabilities. The evaluation may be expressed numerically, for example.

[0107] Furthermore, the candidate evaluation process may include a process of extracting three options with high numerical values ​​indicating evaluation from among multiple types of response candidates. If the sound information of the utterance included in the first communication information includes the term, for example, "Please place the bananas," the support device 11 may create three types of options as follows: The three types of options created by the support device 11 may be sound information or text data such as, for example, "Please place the bananas on the table without packaging," "Please place the bananas wrapped in transparent resin film on the table," and "Please place the bananas wrapped in transparent resin film on each of the multiple shelves."

[0108] Furthermore, the three options created by the support device 11 may include, for example, image information such as "a storyboard of a person placing an unwrapped banana on a table," "a storyboard of a person placing a banana wrapped in transparent resin film on a table," and "a storyboard of a person stacking multiple boxes containing bananas on top of each other."

[0109] Furthermore, the candidate evaluation process may include a process of arranging multiple image pieces of information in sequence to form a selection. These multiple image pieces of information may include, for example, "a storyboard for the first step of taking a banana out of the box," "a storyboard for the second step of wrapping the banana taken out of the box in a transparent resin film," and "a storyboard for the third step of placing the banana wrapped in the transparent resin film on a shelf." These image pieces of information are arranged in the order of the first, second, and third steps of the work procedure included in the sound information. These image pieces of information may also be created and arranged in the order of the stages of the story included in the sound information, for example, the order of introduction, development, turning point, and conclusion.

[0110] Selection guidance processing is a process that simplifies and rephrases the wording in the three extracted options to make them easier for neurotypical individuals or those with developmental disabilities to choose from. Furthermore, selection guidance processing may also consider emotional analysis of the auditory information and tone analysis of the voice contained in the communication information.

[0111] The technology of processing sound information with a computer to analyze human emotions is a known technology, as described in Japanese Patent Publication No. 6766675, Japanese Patent Publication No. 6524049, Japanese Patent Publication No. 2018-97029, etc., so a detailed explanation will be omitted. The technology of analyzing the tone of sound information with a computer is a known technology, as described in Japanese Patent Publication No. 6192126, Japanese Patent Publication No. 2016-126587, Japanese Patent Publication No. 2008-181355, etc., so a detailed explanation will be omitted.

[0112] The response candidate sorting process evaluates three options based on criteria such as relevance, grammar, and conciseness, and then determines, for example, the order in which to present them to the healthy user terminal 12 or the disabled user terminal 13, based on the user profile, the history of options previously approved by the same healthy user A1 or the same disabled user B1, etc., and optimizes the order in which they are presented.

[0113] The presentation optimization process includes analyzing the entire conversation contained in the audio information and estimating the intentions of either a healthy person A1 or a person with a disability B1. Furthermore, the presentation optimization process includes converting the three options into an appropriate format, such as image information including a storyboard, audio information, and text data, based on the intention estimation results, storyboard text data, user profile, and past optimization history. The storyboard may be in a comic book style, illustration style, line drawing style, etc.

[0114] The selection response processing is a process that analyzes the expression format, vocabulary level, output mode, etc., included in the options previously approved by a healthy person A1 or a person with a disability B1, and writes the results to the user profile.

[0115] Thus, the support device 11 creates the first converted information by performing a process to clarify unclear information contained in the first transmitted information, so that the image information, sound information, text data, etc. contained in the first converted information are objectively consistent with the intentions of healthy person A1. Furthermore, in step S32, the support device 11 creates the current first converted information by referring to past first converted information that has a history of being approved by the same healthy person A1 as the current healthy person A1 from among the history of numerous first converted information stored in the auxiliary storage device 35, so that it is objectively consistent with the intentions of the current healthy person A1.

[0116] The support device 11 may store the process and results of the processing performed in step S32 in an auxiliary storage device. The support device 11 may use the process and results of the processing performed in step S32 to train a learning model using machine learning. The support device 11 may transmit the first conversion information created in step S32 to the healthy person terminal 12 in step S33.

[0117] When the healthy person terminal 12 obtains the first conversion information from the support device 11 in step S11, the healthy person A1 confirms the first conversion information. The first conversion information may include multiple types, for example, three types of choices. In step S12, the healthy person A1 may decide whether the intention of the utterance he made this time is consistent with the information contained in the first conversion information. Even if the first conversion information presented by the support device 11 this time was created by referring to the history of first conversion information previously approved by the healthy person A1, if the conditions in the predetermined environment, such as rules, work procedures, products handled, etc., have changed since then, the first conversion information presented this time may not match the current conditions of the predetermined environment.

[0118] Then, if healthy person A1 determines "Yes" in step S12, they may proceed to step S13 and perform an operation to approve the first conversion information on the healthy person terminal 12. Also, if the first conversion information contains three options, and healthy person A1 determines "Yes" in step S12 because they judge that any of the options included in the first conversion information is consistent with the intent of the utterance they made, they may perform an operation to approve the option they judged to be consistent in step S13.

[0119] On the other hand, if healthy person A1 determines No in step S12, they may perform the operation in step S14 to input a request for correction to the first conversion information. The request for correction to the first conversion information may include, for example, the following: "Please place the cardboard box containing the bananas on the table."

[0120] When the healthy user terminal 12 proceeds to step S15 via step S13, it may transmit the input content performed in step S13 to the support device 11. When the healthy user terminal 12 proceeds to step S15 via step S14, it may transmit the input content performed in step S14 to the support device 11. In addition, the healthy user terminal 12 may store the processing results performed from step S10 to step S15 in the auxiliary storage device 19.

[0121] When the support device 11 obtains the information transmitted from the healthy person's terminal 12 in step S15 described above, it may determine in step S34 whether the first conversion information has been approved by the healthy person's terminal 12. If the support device 11 determines Yes in step S34, it may proceed to step S35 and transmit the first conversion information to the disabled person's terminal 13.

[0122] If the support device 11 determines "No" in step S34, it may proceed to step S36 to modify the first conversion information and then proceed to step S33. If the support device 11 receives a modification request from the healthy person terminal 12, "Please place the cardboard box containing the bananas on the table," the support device 11 may modify the sound information, image information, and text data included in the first conversion information in step S36 based on the modification request.

[0123] The support device 11 may store various information corresponding to the processing results from steps S31 to S36 in the auxiliary storage device 35. In this case, the various information stored in the auxiliary storage device 35 may be stored in association with the date and time of processing, the healthy person account, and the disabled person account.

[0124] As described above, the support device 11 can implement the communication support method shown in Figure 4, that is, the processing from steps S31 to S36 in Figure 4, by operating the program stored in the program storage unit 50.

[0125] The disability terminal 13 used by the disabled person B1 may acquire the first conversion information transmitted from the support device 11 in step S35 in step S50. The disabled person B1 can confirm the first conversion information using the display 28E and output device 29 of the disability terminal 13.

[0126] (Example of processing performed by the communication support system 2) Figure 5 shows an example of processing 2 performed by the communication support system 10. Processing Example 2 corresponds to the flow from step S50 onwards in Figure 4. In other words, it is an example in which the response from disabled person B1, who received a request from able-bodied person A1, is transmitted to able-bodied person A1 via the support device 11. Disabled person B1 may also have the disabled person's terminal 13 access the website provided by the support device 11 on the network 15 in step S51 and input the second transmission information into the disabled person's terminal 13.

[0127] The second communication information is information that the disabled person B1, who has confirmed the first conversion information, intends to convey to the able-bodied person A1. The second communication information may include, for example, audio information including speech made by the disabled person B1, image information used by the disabled person B1 when they made the speech, and text data. The content of the speech may include greetings, requests, inquiries, confirmations, replies, and answers.

[0128] Person B1 with a disability may, as a response to a request received from person A1 without a disability, input the following as second communication information into the disabled person's terminal 13: for example, audio information and text data such as "I placed a banana," and image information including a photo of a banana. Then, in step S51, person B1 with a disability may transmit the second communication information from the disabled person's terminal 13 to the assistive device 11.

[0129] In step S37, the support device 11 may acquire second communication information from the disabled terminal 13 and store the second communication information in the auxiliary storage device 35. Furthermore, in step S37, the support device 11 may preprocess the second communication information. The preprocessing performed by the support device 11 is a function for estimating the intention of the disabled person B1 contained in the second communication information. The outline of the preprocessing performed by the support device 11 is the same as the outline of the preprocessing performed by the support device 11 in step S31 of Figure 4. In other words, the artificial intelligence unit 44 has a function to determine whether the second communication information is clear or unclear by cooperating with the auxiliary storage device 35.

[0130] The artificial intelligence unit 44 of the support device 11 may create second converted information in step S38 based on the results of preprocessing the second transmission information. The outline of the process by which the artificial intelligence unit 44 of the support device 11 creates the second converted information in step S38 is the same as the outline of the process for creating the first converted information in step S32. For example, the process for creating the second converted information may include a process for amplifying the volume of sound information that is difficult to hear in the sound information included in the second transmission information to a volume exceeding a predetermined value.

[0131] Furthermore, the process for creating the second translation information may include a process for converting a language not used in the listener's residential area into a language used in the listener's residential area. In addition, the process for creating the second translation information may include a process for removing noise contained in the sound information. Furthermore, the process for creating the second translation information may include processes such as candidate generation, candidate evaluation, selection guidance, response candidate sorting, presentation optimization, and selection response processing.

[0132] Furthermore, the candidate evaluation process performed by the artificial intelligence unit 44 of the support device 11 when creating the second conversion information in step S38 may include a process of extracting three options with high numerical values ​​indicating evaluation from among multiple types of response candidates. If the sound information of the utterance included in the second communication information includes the term, for example, "I placed a banana.", the support device 11 may create three types of options as follows: The three types of options created by the support device 11 may be sound information or text data such as, for example, "I placed a banana on the table without packaging.", "I placed a banana wrapped in transparent resin film on the table.", and "I placed bananas wrapped in transparent resin film on multiple shelves."

[0133] Furthermore, the three options created by the support device 11 may include, for example, image information such as "a storyboard showing an unpackaged banana placed on a table," "a storyboard showing a banana wrapped in transparent resin film placed on a table," and "a storyboard showing bananas stored on multiple shelves, with the multiple shelves stacked vertically."

[0134] Thus, when the support device 11 performs a process to clarify unclear information contained in the second transmitted information and creates the second converted information, it is presumed that the image information, sound information, text data, etc. contained in the second converted information are objectively consistent with the intentions of the disabled person B1. Furthermore, when the support device 11 creates the current second converted information in step S32 by referring to past second converted information that has a history of being approved by the same disabled person B1 as the current disabled person B1 from among the history of numerous second converted information stored in the auxiliary storage device 35 in step S38, it is presumed that this will objectively be consistent with the intentions of the disabled person B1 in the current case.

[0135] The support device 11 may store the process and results of the processing performed in step S38 in the auxiliary storage device 35. The support device 11 may use the process and results of the processing performed in step S38 to train a learning model using machine learning. The support device 11 may transmit the second conversion information created in step S38 to the disabled terminal 13 in step S39.

[0136] When the disabled person terminal 13 obtains the second conversion information from the assistive device 11 in step S52, the disabled person B1 confirms the second conversion information. The second conversion information may include multiple types, for example, three types of options. In step S53, the disabled person B1 may determine whether the intent of the utterance he made matches the information contained in the second conversion information. Even if the second conversion information presented by the assistive device 11 was created by referring to the history of second conversion information previously approved by the disabled person B1, if the conditions in the predetermined environment, such as rules, work procedures, or products handled, have changed since then, the second conversion information presented this time may not match the current conditions of the predetermined environment.

[0137] Then, if disabled person B1 determines "Yes" in step S53, they may proceed to step S54 and perform an operation to approve the second conversion information on the disabled person's terminal 13. Also, if the second conversion information contains three types of options, and disabled person B1 determines "Yes" in step S53 because they judge that any of the options included in the second conversion information is consistent with the intent of the utterance they made, they may perform an operation to approve the option they judged to be consistent in step S54.

[0138] On the other hand, if person B1 with a disability determines "No" in step S53, they may perform the operation of inputting a request for correction to the second conversion information in step S55. The request for correction to the second conversion information may include, for example, the following: "I placed the cardboard box containing the bananas on the table."

[0139] If the disabled terminal 13 proceeds to step S56 after passing through step S54, it may transmit the input content performed in step S54 to the support device 11. If the disabled terminal 13 proceeds to step S56 after passing through step S55, it may transmit the input content performed in step S56 to the support device 11. In addition, the disabled terminal 13 may store the processing results performed from step S50 to step S56 in the auxiliary storage device 27.

[0140] When the support device 11 obtains the information transmitted from the disabled terminal 13 in step S56 described above, it may determine in step S40 whether the second conversion information has been approved by the disabled terminal 13. If the support device 11 determines Yes in step S40, it may proceed to step S41 and transmit the second conversion information to the able-bodied terminal 12.

[0141] If the support device 11 determines No in step S40, it may proceed to step S42 to modify the second conversion information and then proceed to step S39. If the support device 11 receives a modification request from the disabled terminal 13, "I placed the cardboard box containing bananas on the table," the support device 11 may modify the sound information, image information, and text data included in the second conversion information in step S42 based on the modification request.

[0142] The support device 11 may store various information corresponding to the processing results from steps S37 to S42 in the auxiliary storage device 35. In this case, the various information stored in the auxiliary storage device 35 may be stored in association with the date and time of processing, the healthy person account, and the disabled person account.

[0143] As described above, the support device 11 can implement the communication support method shown in Figure 5, that is, the processing from steps S37 to S42 in Figure 5, by operating the program stored in the program storage unit 50.

[0144] Furthermore, the healthy person terminal 12 used by healthy person A1 may acquire the second conversion information transmitted from the support device 11 in step S41 in step S16.

[0145] (Processing Example 3) Processing Example 3 is an example in which, due to an abnormal communication environment, a healthy person's terminal 12 and a disabled person's terminal 13 communicate with each other via a network 15 consisting of short-range wireless communication, etc., without going through the assistive device 11. In this processing example 3, the healthy person's terminal 12 performs the processes from steps S10 to S15 shown in Figure 4, as well as the processes from steps S31 to S36 shown in Figure 4.

[0146] It is assumed that the healthy user terminal 12 has the same configuration and functions as the support device 11, for example, a processor 33 and an auxiliary storage device 35, in order to perform the processes from steps S31 to S36 shown in Figure 4. In other words, the program stored in the auxiliary storage device 35 of the healthy user terminal 12 is executed, and the processes from steps S30 to S35 in Figure 4 are performed.

[0147] Therefore, the healthy person terminal 12 can perform preprocessing using the first transmission information and create the first conversion information. Healthy person A1 checks the first conversion information and inputs an approval or correction request. When healthy person A1 requests a correction to the first conversion information, the healthy person terminal 12 can correct the first conversion information. Furthermore, the healthy person terminal 12 may transmit the first conversion information approved by healthy person A1 to the disabled person terminal 13 in step S35.

[0148] (Processing Example 4) Processing Example 4 is an example in which, due to an abnormal communication environment, a healthy person's terminal 12 and a disabled person's terminal 13 communicate with each other via a network 15 consisting of short-range wireless communication, etc., without going through the assistive device 11. In this processing example 4, the disabled person's terminal 13 performs the processes from steps S50 to S56 shown in Figure 5, as well as the processes from steps S37 to S41 shown in Figure 5.

[0149] It is assumed that the disabled terminal 13 has the same configuration and functions as the assistive device 11, for example, a processor 33 and an auxiliary storage device 35, in order to perform the processes from steps S37 to S41 shown in Figure 5. In other words, the program stored in the auxiliary storage device 35 of the disabled terminal 13 is executed, and the processes from steps S35 to S41 in Figure 5 are performed.

[0150] Therefore, the disabled person terminal 13 performs preprocessing using the second transmission information and creates the second conversion information. Disabled person B1 checks the second conversion information and inputs an approval or correction request. When disabled person B1 requests a correction to the second conversion information, the disabled person terminal 13 corrects the second conversion information. Furthermore, in step S41, the disabled person terminal 13 transmits the second conversion information approved by disabled person B1 to the able-bodied person terminal 12.

[0151] (Effects of this embodiment) The communication support system 10 of this embodiment can send first converted information, which includes image information consistent with the intention of a healthy person A1 contained in the audio of the first communication information, to the disabled person's terminal 13, or send second converted information, which includes image information consistent with the intention of a disabled person B1 contained in the audio of the second communication information, to the healthy person's terminal 12. Therefore, the communication support system 10 has improved functionality in supporting communication between a healthy person A1 and a disabled person B1.

[0152] Furthermore, the support device 11, or the healthy person's terminal 12, sends the first conversion information approved by the healthy person A1 to the disabled person's terminal 13. In addition, the support device 11, or the disabled person's terminal 13, sends the second conversion information approved by the disabled person B1 to the healthy person's terminal 12. Therefore, the support device 11, or the healthy person's terminal 12, or the disabled person's terminal 13, improves its ability to support communication between the healthy person A1 and the disabled person B1.

[0153] Furthermore, if the support device 11, the healthy person terminal 12, or the disabled person terminal 13 determines that the meaning of the information contained in the transmitted information is unclear, it generates image information that clarifies the meaning of the information. Therefore, the support device 11, the healthy person terminal 12, or the disabled person terminal 13 further improves its ability to support communication between healthy person A1 and disabled person B1.

[0154] Furthermore, multiple image pieces are arranged in the order of the conversation between able-bodied person A1 and disabled person B1, or in the order of the tasks performed within the conversation, and presented to the disabled person's terminal 13 or the able-bodied person's terminal 12. Therefore, in the communication support system 10 of this embodiment, able-bodied person A1 or disabled person B1 can more easily understand the other person's intent.

[0155] Furthermore, in the communication support system 10 of this embodiment, the first conversion information provided to the disabled person terminal 13 is set such that the volume of the sound information is higher when the disabled person B1 is hearing impaired than when the disabled person B1 is not hearing impaired. Therefore, the disabled person B1 can more easily understand the speech intent of the able-bodied person A1.

[0156] (supplementary explanation) An example of the technical meaning disclosed in this embodiment is as follows: Communication support device 11 is an example of a communication support device and management computer. Processor 33 is an example of a processor. Healthy person A1 or disabled person B1 is an example of a first target person. Healthy person terminal 12 or disabled person terminal 13 is an example of a first target person terminal. Disabled person terminal 13 or healthy person terminal 12 is an example of a second target person terminal.

[0157] Step S31 or Step S37 is an example of the first process. Step S32 or Step S38 is an example of the second process. Step S35 or Step S41 is an example of the third process. Steps S33, S34, S36 or S39, S40, and S42 are examples of the fourth process. Step S32 or Step S38 is an example of the fifth process.

[0158] A healthy person A1 or a person with a disability B1 is an example of a sender. A person with a disability B1 or a healthy person A1 is an example of a receiver. A healthy person terminal 12 or a person with a disability terminal 13 is an example of a sending computer. A person with a disability terminal 13 or a healthy person terminal 12 is an example of a receiving computer. Input device 20 or input device 28 is an example of an input device. Processor 17 or processor 25 is an example of a processor. Step S31 or step S37 is an example of a first process. Step S28 or step S38 is an example of a second process. Step S35 or step S41 is an example of a third process. The first transmitted information and the second transmitted information are examples of transmitted information. The first converted information and the second converted information are examples of converted information.

[0159] In this embodiment, the predetermined environment in which the first subject and the second subject communicate may be a workplace, a hospital, while shopping in daily life, an educational setting, an educational setting for young children, etc. Furthermore, both the first subject and the second subject may be able-bodied or both may be disabled.

[0160] Furthermore, the communication support system may be implemented by a single computer or by multiple distributed computers. The communication support system may consist of one or more computers, such as servers, supercomputers, mainframes, servers, workstations, etc.

[0161] This embodiment discloses the following characteristic configuration. For example, a non-temporary storage medium storing a non-temporary program that causes a management computer, which is communicably connected to a first target terminal used by a first target person and a second target terminal used by a second target person, to execute processing, wherein the program is configured to cause the management computer to execute: a first process of acquiring first communication information including the voice of the first target person from the first target person terminal; a second process of creating first conversion information including image information consistent with the intention of the first target person contained in the voice of the first communication information; and a third process of sending the first conversion information to the second target terminal.

[0162] Furthermore, the non-temporary storage medium is operated by a transmitting computer that has an input device for the sender to input transmission information and is communicably connected to a receiving computer used by the receiver with whom the sender communicates, and stores a non-temporary program that causes the transmitting computer to execute processing, wherein the program is configured to cause the transmitting computer to execute a first process of acquiring the transmission information input by the input device, a second process of creating converted information including image information consistent with the meaning of the voice contained in the transmission information, and a third process of sending the converted information to the receiving computer. Note that the program can also be understood as a program product. [Industrial applicability]

[0163] This embodiment can be used as a communication support device and program to facilitate communication between individuals. [Explanation of Symbols]

[0164] 10...Communication support device, 12...Terminal for able-bodied individuals, 13...Terminal for people with disabilities, 17, 25, 33...Processor, 20, 28...Input device, A1...Able-bodied individuals, B1...People with disabilities

Claims

1. A communication support device having a processor that performs processing, and being communicably connected to a first target terminal used by a first target and a second target terminal used by a second target, wherein the processor is configured to perform a first process of acquiring transmission information including the voice of the first target from the first target terminal, a second process of creating converted information including image information consistent with the intent of the first target contained in the voice of the transmission information, and a third process of sending the converted information to the second target terminal, wherein the second process performed by the processor includes a process of determining whether the meaning of the information contained in the transmission information is unclear, and if it is determined that the meaning of the information is unclear because the transmission information contains information lacking one or more of the 5W2H, the process of generating image information that clarifies the meaning of the information, and before performing the third process, the processor sends the converted information to the first target terminal and, as a result of the first target determining that their intent and the converted information are consistent, approves sending the converted information to the second target terminal. A communication support device configured to perform a fourth process in which approval information indicating that the first target person has determined that the conversion information is inconsistent with their own intentions is obtained from the first target person's terminal, or correction request information indicating a request for correction of the conversion information as a result of the first target person determining that the conversion information is inconsistent with their own intentions; the third process performed by the processor is configured to send the conversion information to the second target person's terminal only if the approval information is obtained in the fourth process; the processor is configured to perform a fifth process in which, if the approval information is not obtained in the fourth process but the correction request information is obtained, the conversion information is not sent to the second target person's terminal, the conversion information is corrected based on the correction request information, and the corrected conversion information is sent back to the first target person's terminal; the processor is configured to perform the fourth process again with respect to the conversion information sent back to the first target person's terminal by the fifth process; and the third process performed by the processor is configured to send the corrected conversion information to the second target person's terminal only if the approval information is obtained in the fourth process performed again.

2. A communication support device according to claim 1, wherein the correction request information includes the content of the correction of the conversion information entered into the first target terminal by the first target.

3. A communication support device according to claim 1, wherein the conversion information created by the processor includes a plurality of image information arranged in the order of the beginning, development, turning point, and conclusion of a plurality of steps in a work procedure or story contained in the audio of the transmitted information.

4. A communication support method performed by a management computer that is communicably connected to a first target terminal used by a first target person and a second target terminal used by a second target person, wherein the management computer performs a first process of acquiring transmission information including the voice of the first target person from the first target terminal; a second process of creating converted information including image information consistent with the intent of the first target person contained in the voice of the transmission information; and a third process of sending the converted information to the second target terminal, wherein the second process performed by the management computer includes a process of determining whether the meaning of the information contained in the transmission information is unclear, and if it is determined that the meaning of the information is unclear because the transmission information contains information lacking one or more of the 5W2H, then a process of generating image information that clarifies the meaning of the information, and before performing the third process, the management computer sends the converted information to the first target terminal and accepts that the first target person will send the converted information to the second target terminal as a result of determining that their intent and the converted information are consistent. A communication support method comprising: a fourth process is performed to obtain approval information indicating that the conversion information has been acknowledged, or correction request information indicating a request for correction of the conversion information as a result of the first target person judging that their intention and the conversion information are inconsistent; the third process performed by the management computer sends the conversion information to the second target person terminal only if the approval information is obtained in the fourth process; if the management computer does not obtain the approval information but obtains the correction request information in the fourth process, it does not send the conversion information to the second target person terminal, but corrects the conversion information based on the correction request information and performs a fifth process to send the corrected conversion information back to the first target person terminal; the management computer then performs the fourth process again with respect to the conversion information sent back to the first target person terminal by the fifth process; and the third process performed by the management computer sends the corrected conversion information to the second target person terminal only if the approval information is obtained in the fourth process performed again.

5. A non-temporary program that causes a management computer, which is connected to a first target terminal used by a first target person and a second target terminal used by a second target person, to execute processing, wherein the management computer is configured to execute: a first process of acquiring communication information including the voice of the first target person from the first target person terminal; a second process of creating converted information including image information consistent with the intention of the first target person contained in the voice of the communication information; and a third process of sending the converted information to the second target terminal, wherein the second process to be executed by the management computer includes a process of determining whether the meaning of the information contained in the communication information is unclear, and if it is determined that the meaning of the information is unclear because the communication information contains information lacking one or more of the 5W2H, then a process of generating image information that clarifies the meaning of the information, wherein before executing the third process, the management computer is instructed to send the converted information to the first target terminal, and the first target person approves sending the converted information to the second target terminal as a result of determining that their intention and the converted information are consistent. A program configured to execute a fourth process in which the management computer receives approval information indicating that the first target has determined that the conversion information does not match its intent, or correction request information indicating a request to correct the conversion information as a result of the first target determining that the conversion information does not match its intent; the third process to be executed by the management computer is configured to send the conversion information to the second target terminal only if the approval information is received in the fourth process; the fifth process to be executed in which the management computer does not receive the conversion information but receives the correction request information in the fourth process, does not send the conversion information to the second target terminal, corrects the conversion information based on the correction request information, and sends the corrected conversion information back to the first target terminal; the fourth process to be executed again in which the management computer receives the conversion information that has been sent back to the first target terminal by the fifth process; and the third process to be executed by the management computer is configured to send the corrected conversion information to the second target terminal only if the approval information is received in the fourth process executed again.

6. A transmitting computer having an input device for the sender to input transmission information and a processor to perform processing, and being communicably connected to a receiving computer used by a receiver with whom the sender communicates, wherein the processor is configured to perform a first process of acquiring the transmission information input by the input device, a second process of creating converted information including image information consistent with the meaning of the voice contained in the transmission information, and a third process of sending the converted information to the receiving computer, wherein the second process performed by the processor includes a process of determining whether the meaning of the information contained in the transmission information is unclear, and if it is determined that the meaning of the information is unclear because the transmission information contains information lacking one or more of the 5W2H, then a process of generating image information that clarifies the meaning of the information, and before performing the third process, the processor presents the converted information to the sender, and if the sender determines that the converted information is consistent with their intention, the processor sends the converted information to the receiving computer. A sending computer is configured to perform a fourth process in which it obtains approval information indicating that it has approved the conversion, or correction request information indicating a request for correction of the conversion information as a result of the sender determining that the conversion information does not match its own intentions; the third process performed by the processor is configured to send the conversion information to the receiving computer only if it has obtained the approval information in the fourth process; if the processor has obtained the correction request information but has not obtained the approval information in the fourth process, it does not send the conversion information to the receiving computer, but corrects the conversion information based on the correction request information and performs a fifth process in which it presents the corrected conversion information to the sender again; the processor is configured to perform the fourth process again with respect to the conversion information presented to the sender again by the fifth process; and the third process performed by the processor is configured to send the corrected conversion information to the receiving computer only if it has obtained the approval information in the fourth process performed again.

7. A communication support method performed by a transmitting computer having an input device for transmitting information and being communicably connected to a receiving computer used by a receiver with whom the transmitting computer communicates, wherein the transmitting computer performs a first process of acquiring the transmission information input by the input device, a second process of creating converted information including image information consistent with the meaning of the voice contained in the transmission information, and a third process of sending the converted information to the receiving computer, wherein the second process performed by the transmitting computer includes a process of determining whether the meaning of the information contained in the transmission information is unclear, and if it is determined that the meaning of the information is unclear because the transmission information contains information lacking one or more of the 5W2H, the process of generating image information that clarifies the meaning of the information, and before performing the third process, the transmitting computer presents the converted information to the transmitting computer, and as a result of the transmitting computer determining that its intention and the converted information are consistent, the transmitted computer presents the converted information to the receiving computer A communication support method comprising: a fourth process which is performed to obtain approval information indicating that the sender has approved sending the information to the recipient, or correction request information indicating a request for correction of the conversion information as a result of the sender determining that the conversion information does not match their intentions; the third process which is performed by the sending computer sends the conversion information to the receiving computer only if the approval information is obtained in the fourth process; if the sending computer does not obtain the approval information in the fourth process but obtains the correction request information, it does not send the conversion information to the receiving computer, but corrects the conversion information based on the correction request information and performs a fifth process which presents the corrected conversion information to the sender again; the sending computer performs the fourth process again with respect to the conversion information presented to the sender again by the fifth process; and the third process which is performed by the sending computer sends the corrected conversion information to the receiving computer only if the approval information is obtained in the fourth process which is performed again.

8. A non-temporary program that operates on a transmitting computer having an input device in which the sender inputs transmission information, and which is communicably connected to a receiving computer used by a receiver with whom the sender communicates, and which causes the transmitting computer to execute processing, wherein the transmitting computer is configured to execute a first process of acquiring the transmission information input by the input device, a second process of creating converted information including image information consistent with the meaning of the voice contained in the transmission information, and a third process of sending the converted information to the receiving computer, wherein the second process to be executed by the transmitting computer includes a process of determining whether the meaning of the information contained in the transmission information is unclear, and if it is determined that the meaning of the information is unclear because the transmission information contains information lacking one or more of the 5W2H, then a process of generating image information that clarifies the meaning of the information, and before executing the third process, the transmitting computer is instructed to present the converted information to the sender, and as a result of the sender determining that the converted information is consistent with their intention, the converted information is presented to the sender A program configured to execute a fourth process that obtains approval information indicating that the sender has authorized sending the converted information to the receiving computer, or correction request information indicating a request for correction of the converted information as a result of the sender determining that the converted information does not match their intentions; the third process to be executed by the sending computer is configured to send the converted information to the receiving computer only if the approval information is obtained in the fourth process; the fifth process to be executed by the sending computer if, in the fourth process, the approval information is not obtained but the correction request information is obtained, the converted information is not sent to the receiving computer, the converted information is corrected based on the correction request information, and the corrected converted information is presented to the sender again; the fourth process to be executed again by the sending computer with respect to the converted information presented to the sender again by the fifth process; and the third process to be executed by the sending computer is configured to send the corrected converted information to the receiving computer only if the approval information is obtained in the fourth process executed again.

Images