system

Abstract

We provide the system. [Solution] A means of acquiring the voice of a visitor, A conversion means for converting the aforementioned audio into text data, An analysis means that analyzes the aforementioned text data to determine the visitor's requirements, Based on the aforementioned judgment, if the visitor is not suspicious, a means of notifying the resident of the requirements, A means of selecting whether a resident will respond to a visitor according to the aforementioned requirements, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For elderly people living alone or those who need to deal with visitors on a daily basis, fraud by visitors and stress caused by unnecessary sales are serious problems. Also, there is a lack of means to ensure a quick and safe response to visitors, which is also an issue. As a result, the elderly are at risk of missing important visits and cannot feel secure in their daily lives.

Means for Solving the Problems

[0005] The present invention provides a system equipped with acquisition means and conversion means for acquiring visitor voices and converting them into text data. It also includes analysis means for analyzing the text data and evaluating its safety. Based on the analysis results, the system provides a notification means that allows residents to respond with confidence, informing them of the visit details, and further provides a selection means for residents to choose whether or not to respond. This realizes a means to safely manage direct interactions with visitors and prevent harm from suspicious individuals.

[0006] "Acquisition means" refers to a device or method for receiving a visitor's voice via an intercom and converting that data into a format usable within the system.

[0007] A "conversion means" is a mechanism for converting acquired audio data into text data using natural language processing technology.

[0008] "Analysis means" refers to algorithms and processing systems that analyze text data and determine whether a visitor's requirements are safe or suspicious based on their content.

[0009] "Notification means" refers to devices or methods for communicating visitor requirements to residents in text or audio format.

[0010] A "choice mechanism" refers to an interface or method that allows residents to choose whether to allow or deny interaction with visitors.

[0011] "Means of responding" refers to the functions or devices used by residents to directly respond to visitors via intercom.

[0012] A "response control means" is a system or method for controlling the continuation of a visit by playing a pre-set message to a suspicious visitor. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, a numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is an intercom system for automating visitor reception. The system primarily relies on a process of acquiring and analyzing visitor voices to determine the necessary response. Its operation is described below.

[0035] System Overview

[0036] The system is mainly divided into three components: server, terminal, and user. The server primarily handles data processing, the terminal provides the user interface, and the user makes decisions on how to respond based on the system's presentation.

[0037] Program processing

[0038] The server receives the visitor's voice. The system's intercom acquires the visitor's voice in real time and sends that data to the server.

[0039] The server converts speech to text. Using speech recognition technology, it converts the received audio data into text, preparing it for analysis in the next step.

[0040] The server analyzes the text data. The analysis engine evaluates keywords and context within the text and performs filtering to determine whether the visitor's requirements are safe or suspicious.

[0041] The server generates a notification based on the analysis results. Based on the analysis results, it sends a notification to the terminal, informing the user of the visitor's requirements.

[0042] The device displays notifications to the user. For example, the visitor's requirements are displayed on the resident's smartphone or a dedicated terminal inside the room, and audio alerts can also be set up.

[0043] The user chooses to respond based on the notification content. The user can choose to allow or deny the response through their device.

[0044] As an example of this system, if a delivery person arrives to deliver a package, the server recognizes the keyword "delivery" and determines that the visit is safe. As a result, the terminal notifies the user of the package's arrival, and the user can optionally decide whether or not to answer the door. This process protects the user from unnecessary visits and potential dangers. Furthermore, the system records visit history, which can be used to improve the accuracy of decisions regarding subsequent visits.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The server receives the visitor's voice from the intercom microphone. The voice data is converted to a digital format and stored in a form that can be processed internally by the server.

[0048] Step 2:

[0049] The server sends the audio data to the speech recognition engine, which converts it into text data. The speech recognition engine analyzes the content of the audio and outputs it as a string of characters.

[0050] Step 3:

[0051] The server passes the converted text data to the analysis engine, which performs keyword detection and intent evaluation. This helps determine whether the visitor's requirements are safe.

[0052] Step 4:

[0053] The server generates a visitor safety assessment based on the analysis results. If suspicious content is detected, a flag is set to identify the relevant processing result, and a notification preparation state is established.

[0054] Step 5:

[0055] The server transmits the visitor's requirements and analysis results to the terminal. The terminal then informs the user of the situation by displaying the relevant information.

[0056] Step 6:

[0057] The device notifies the user, informing them of the visit via voice alerts or pop-up messages. The user is then given the option to choose whether or not to respond.

[0058] Step 7:

[0059] The user chooses to allow or deny the interaction via their device. If they choose to allow, they are able to communicate directly with the visitor through the intercom.

[0060] Step 8:

[0061] The terminal feeds back the response result to the server based on the user's selection. If the request is rejected, the server plays a pre-configured response message to the visitor.

[0062] Step 9:

[0063] The server logs visit details and user interaction results in a database, which is used to improve the accuracy of future analysis and decision-making.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] Current intercom systems sometimes struggle to accurately identify visitors and provide residents with timely and appropriate information. Furthermore, measures against suspicious individuals and management of visitor history are insufficient. This creates unnecessary inconvenience and exposes residents to potential dangers, thus creating a need for a system that addresses these issues.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes acquisition means for acquiring the visitor's voice, conversion means for converting the voice into text data, analysis means for analyzing the text data to determine the visitor's requirements, notification means for notifying the resident of the requirements, and means for recording the history. This makes it possible to quickly provide visitor information to the resident and manage the visit history.

[0069] "Acquisition means" refers to a device or method that has the function of capturing audio from visitors in real time and transmitting it to a server as digital data.

[0070] "Conversion means" refers to a process that includes speech recognition technology used to convert acquired audio data into an analyzable text format.

[0071] "Analysis means" refers to a system or procedure that uses natural language processing technology to extract the intentions and requirements of visitors from text data and evaluate their content.

[0072] "Notification means" refers to a method and apparatus for appropriately presenting analyzed visitor information to residents and generating and transmitting messages to encourage necessary actions.

[0073] "Selection method" refers to an interface and process that allows residents to intuitively choose response options based on the visitor information presented.

[0074] "Means" is a comprehensive term that refers to the devices and methods used to achieve a specific function within a system.

[0075] As an embodiment of this invention, an intercom system for automating visitor reception will be described. This system mainly consists of three components: a server, a terminal, and a user.

[0076] Servers are the core of data processing. Computer servers are used as hardware to receive visitor voice data in real time and process that information. To convert the voice into digital text, speech recognition software such as Google® Speech-to-Text API and Azure® Speech Service is used. This converted text data is then analyzed by an analysis engine, which uses natural language processing technology to analyze keywords and context, revealing the visitor's requirements and intentions. Based on the analysis results, a safety assessment is made.

[0077] The terminal serves as an interface for the user. Smartphones and dedicated indoor terminals are used to provide residents with visitor information. Notifications are not only displayed on the screen but also communicated to the user via voice and vibration, enabling quick and appropriate responses in an instant.

[0078] Users choose how to respond to visitors based on the information provided by their device. For example, in the case of a delivery service visit, a user who has received a notification that "package has arrived" can immediately decide whether to allow the visitor to enter the room. This system also has a function to record visit history, and analyzing visitor information later contributes to further improving accuracy.

[0079] An example of a prompt would be, "Develop an AI model that explains how to assess the safety of a visit and generate an appropriate notification for the user when a visitor says, 'It's a delivery.'" In this way, generative AI models can be used to build more sophisticated response logic.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] The server acquires the visitor's voice. The intercom sends the voice data received in real time to the server. This input voice data is stored in the server's storage and used as material for conversion processing.

[0083] Step 2:

[0084] The server converts audio data into text data. Using a speech recognition engine, the server analyzes the audio data and generates corresponding text data. In this conversion process, the speech recognition model analyzes the audio patterns and converts them into text as language. The output is the visitor's spoken content in text format.

[0085] Step 3:

[0086] The server analyzes the text data. Using natural language processing techniques, the server analyzes the meaning of the text. This process involves keyword extraction and contextual analysis to identify the visitor's intent. The output includes visitor requirements and a safety assessment.

[0087] Step 4:

[0088] The server generates a notification based on the analysis results. Based on the analyzed data, the server creates a notification message and sends it to the device. This notification includes visitor requirements and recommended actions regarding the response. The output is the notification message to the user.

[0089] Step 5:

[0090] The terminal displays notifications to the user. The interface on the terminal visually and audibly presents notifications received from the server to the user. This allows the user to check visitor information and take necessary actions immediately. The input is the notification message from the server, and the output is the interface display presented to the user.

[0091] Step 6:

[0092] The user selects a response based on the notification. Based on the information displayed on the device, the user decides how to respond to the visitor. In this step, the user chooses whether to allow or deny the response from the provided options, and this selection is immediately reflected in the system. The input is the content of the notification the user saw on the device, and the output is the response the user selected.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] Dealing with visitors in a residential setting can increase security risks and complexity due to the inclusion of external factors that may infringe upon residents' lives. In particular, difficulties in dealing with suspicious individuals can threaten residents' safety. Therefore, a system is needed that can quickly and accurately assess the safety of visitors and notify residents, enabling appropriate responses.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes acquisition means for acquiring the visitor's voice and image, conversion means for converting the voice into text data, and analysis means for analyzing the text data and image data to determine the visitor's requirements and safety. This makes it possible to quickly and accurately determine the visitor's safety and notify the resident of the results.

[0098] "Acquisition means" refers to a device or method for collecting the audio and images of visitors.

[0099] "Conversion means" refers to a technique or method for converting acquired audio data into text data.

[0100] "Analysis means" refers to a process or system that analyzes audio and image data to evaluate the visitor's requirements and safety.

[0101] "Control means" refers to a method or device for notifying residents based on analysis results and for coordinating and managing communication with visitors.

[0102] A "means of choice" is a method or device that provides a function allowing residents to choose how they respond to visitors.

[0103] The "response function" is a feature that automatically initiates communication with visitors deemed safe and informs residents of the situation.

[0104] To implement this invention, first, a terminal equipped with a camera and microphone is used as an acquisition means for acquiring the visitor's voice and image. The voice data acquired by the terminal is sent to a server, where a conversion means converts the voice into text data. For example, speech recognition technology such as the Google Cloud Speech-to-Text API can be used.

[0105] The server analyzes the converted text and image data using analytical tools. Keyword extraction and contextual analysis techniques are employed to evaluate visitor requirements and security. Efficient data processing is achieved by utilizing cloud-based analytical services such as AWS® Lambda.

[0106] Based on the analysis results, if the control system determines that the visitor is safe, it sends a notification to the resident's smartphone. This notification is used as information for the resident to choose how to respond to the visitor. The selection function allows the resident to decide whether to allow or deny the response and express their intention through their device.

[0107] As a concrete example, residents can initiate direct communication with visitors who are deemed safe. An example of a prompt message might be: "We offer a service that uses the visitor response system to analyze voice data and assess the safety of visitors. Please tell me how to convert speech to text using the Google Cloud Speech-to-Text API." This system makes it easier to deal with suspicious individuals and can improve the security and convenience of residents.

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The device acquires the visitor's voice and image.

[0111] The input is the visitor's real-time audio and video, which is collected by the microphone and camera built into the terminal. The output includes means for generating the acquired audio and image data and preparing it for transmission to the server. Specifically, the terminal detects the visitor's movements and speech and automatically starts recording video and audio.

[0112] Step 2:

[0113] The server converts the audio data into text data.

[0114] The system receives audio data sent from a terminal as input and generates text data as output. This conversion utilizes a speech recognition API (e.g., Google Cloud Speech-to-Text) to perform data calculations that extract string information from the audio waveform. Specifically, the system temporarily stores the audio data on a server and then calls the API to request recognition processing.

[0115] Step 3:

[0116] The server analyzes the text and image data.

[0117] The input consists of text data obtained through speech conversion and initial image data, which are used to evaluate the visitor's requirements and safety. Keyword extraction and machine learning algorithms are applied to the analysis to detect suspicious individuals and estimate the purpose of their visit. Specifically, the analysis engine on the server parses the text and applies image recognition technology to perform facial recognition.

[0118] Step 4:

[0119] Based on the analysis results, the server sends a notification to the terminal via the control system.

[0120] The system takes the analysis results (e.g., whether the visitor is safe) as input and prepares a notification to be output to the resident's device based on that. The notification includes visitor information and the safety assessment result. Specifically, the server constructs the notification message and sends it to the device as a push notification.

[0121] Step 5:

[0122] The user can choose whether or not to respond to visitors through their device.

[0123] The system takes notifications from the device as input and outputs a choice to allow or deny the interaction. Specifically, the device's user interface presents the resident with options, and the user makes a decision through buttons on the screen. This process also allows the user to warn about visitors deemed unsafe.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention is an intercom system that combines an emotion engine with a visitor reception system to enable responses that take into account the emotional state of the resident. It is mainly performed through three components: a server, a terminal, and a user.

[0126] System Overview

[0127] This system is based on a process of acquiring and analyzing visitors' voices. Furthermore, it can recognize the user's emotional state in real time and adjust its response to visitors based on that information.

[0128] Program processing

[0129] The server acquires the visitor's voice through the intercom. This voice data is then converted into text data using speech recognition technology and analyzed.

[0130] The server analyzes the text data and evaluates the visitor's requirements. It determines whether the visitor is safe or suspicious and passes the result on to the next processing step.

[0131] The device acquires the resident's emotions using an emotion engine. The emotion engine analyzes features such as voice tone, facial expressions, and volume to understand the emotional state in real time.

[0132] The server adjusts response actions based on data from the emotion engine. If a resident is stressed, it enhances automated response messages and softens visitor interactions to reduce stress.

[0133] The terminal presents residents with pre-configured responses and recommends the most appropriate way to respond to visitors. This includes options such as allowing, postponing, or declining the response.

[0134] Specific example

[0135] For example, if a visitor is found to be a salesperson, the server can analyze that information in real time, and if the emotion engine detects anxiety or stress in the resident, it can be configured to automatically refuse service. In the case of a delivery person, if the emotion engine recognizes that the resident is relaxed, a recommendation to allow direct service will be displayed on the terminal.

[0136] This invention allows residents to interact with visitors with psychological peace of mind, reducing daily stress and preventing harm from suspicious individuals.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] The server acquires the visitor's voice from the intercom microphone. This audio data is converted to a digital format and stored on the server.

[0140] Step 2:

[0141] The server sends the acquired audio data to the speech recognition engine, which converts the audio into text data. The text data is then used in the subsequent analysis process.

[0142] Step 3:

[0143] The server analyzes text data to determine the visitor's requirements. It detects specific keywords and phrases and performs safety and suspicion assessments.

[0144] Step 4:

[0145] The device detects the user's emotional state using an emotion engine. The emotion engine analyzes the user's voice tone, facial expressions, and movements to evaluate their current emotions in real time.

[0146] Step 5:

[0147] The server receives the evaluation results from the emotion engine and decides on a response action based on the analysis results. For example, if the user is in a stressed state, an automated response message is prepared, and the server is ready to respond flexibly to the visitor.

[0148] Step 6:

[0149] The device displays the visitor's requirements and suggested responses to the user. The user is then given the option to accept, postpone, or decline the interaction.

[0150] Step 7:

[0151] The device will respond to the intercom based on the user's selected response method. If the user allows the call, conversation mode will be enabled; if the user chooses not to answer, a pre-set message will be played.

[0152] Step 8:

[0153] The server records visit details and response results in a database. This record is used for future visit analysis and to improve the accuracy of sentiment recognition.

[0154] (Example 2)

[0155] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0156] The present invention aims to reduce the psychological burden on residents when dealing with visitors and to enable safer and smoother visitor interactions. It also aims to prevent harm from suspicious individuals while enabling flexible responses that take into account the emotional state of residents.

[0157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0158] In this invention, the server includes: an audio acquisition means for acquiring the voice of a visitor; an audio conversion means for converting the voice into text data; a content evaluation means for analyzing the text data to determine the visitor's requirements; an emotion acquisition means for recognizing the emotions of a resident; a response adjustment means for adjusting the response action based on the resident's emotional state acquired by the emotion acquisition means; and a selection presentation means for presenting the adjusted response content to the resident and enabling them to choose how to respond to the visitor. This makes it possible to adjust responses considering the resident's emotional state and to take appropriate measures to prevent harm from suspicious individuals.

[0159] "Voice acquisition means" refers to a device or process for recognizing and recording voices emitted by visitors.

[0160] "Acoustic conversion means" refers to a device or process for analyzing acquired audio data and converting it into a corresponding text data format.

[0161] A "content evaluation tool" is a device or process for analyzing converted text data to determine what the visitor wants.

[0162] An "emotion acquisition method" is a device or process for recognizing emotions by analyzing elements such as voice and facial expressions in order to evaluate the psychological state of a resident.

[0163] A "response adjustment mechanism" is a device or process that adjusts the method of responding to visitors and the messages they send based on the perceived emotional state of the residents.

[0164] A "selection presentation means" is a device or process that presents residents with pre-arranged response options and provides them with multiple choices on how to respond to visitors.

[0165] This invention is a system for handling visitors, and is primarily composed of three components: a server, a terminal, and a user. Its main objective is to acquire and analyze visitors' voices, enabling flexible responses that take into account the emotional state of the resident. By combining voice recognition technology and an emotion recognition engine, it achieves appropriate responses to visitors.

[0166] The server collects visitor voices from the intercom as a means of acquiring audio. The collected audio data is converted into text data using commercially available speech recognition software as a means of audio conversion. This process makes it possible for the user to visualize the visitor's intentions and requests.

[0167] The server then uses the converted text data to employ natural language processing techniques as a content evaluation tool, analyzing the visitor's requirements and intentions. This allows it to determine whether the visitor is safe or suspicious.

[0168] The device provides an emotion acquisition method that uses emotion recognition software to evaluate the emotions of residents. This allows for real-time analysis of residents' voice tone, facial expressions, volume, etc., enabling an understanding of their psychological state.

[0169] The server also includes a response adjustment mechanism that adjusts response actions according to the resident's emotional state, and recommends appropriate responses to provide flexible support if the resident is experiencing stress.

[0170] The system provides a selection tool via a terminal that displays adjusted response options to the user, allowing the user to choose a response policy for the visitor based on this information. These options include allowing, postponing, or not responding.

[0171] As a concrete example, consider a case where a visitor says, "Hello, delivery person." If the emotion acquisition system determines that the user is relaxed, a recommendation to "allow interaction" will be displayed on the device. An example of the prompt text used in this case is as follows: "The system analyzes the visitor's voice and suggests an appropriate response method based on the resident's emotional state. If the visitor's voice says 'Hello, delivery person,' the system will allow interaction with the visitor when the resident is relaxed."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] The server acquires the visitor's voice via the intercom. This acquired voice data is used as input. Specifically, the server receives the voice signal from the intercom's microphone in real time and stores it in digital format. The output at this stage is the digitized voice data.

[0175] Step 2:

[0176] The server uses speech recognition technology to convert this audio data into text data. The input is the previously acquired digital audio data. Specifically, the server uses speech recognition software to analyze the audio waveform and convert it into corresponding text. The output of this process is text data representing what the visitor said.

[0177] Step 3:

[0178] The server analyzes the converted text data to determine the visitor's requirements. The input is text data. Specifically, the server applies natural language processing techniques to analyze the content of the text data, understand its meaning, and identify what the visitor is looking for (e.g., delivery person, salesperson). The output of this process is an evaluation result indicating the visitor's requirements.

[0179] Step 4:

[0180] The device acquires the resident's emotions using emotion recognition technology. The input is data such as the resident's voice and facial expressions. Specifically, the device uses an emotion recognition engine to analyze the resident's voice tone and facial expressions to determine their emotional state, such as whether they are relaxed or stressed. The output of this process is data representing the resident's emotional state.

[0181] Step 5:

[0182] The server adjusts its response actions to visitors based on the resident's emotional state. Inputs include resident emotional state data and visitor requirements assessment results. Specifically, the server customizes response messages and methods, adjusting the response to mitigate stress if the resident is experiencing it. The output of this process is the adjusted response.

[0183] Step 6:

[0184] The terminal presents the user with pre-configured response options, allowing them to select the most appropriate response. The input is the pre-configured response content. Specifically, the terminal presents the resident with options such as "Allow response," "Postpone response," and "Do not respond." The user can then choose the most appropriate option. The output of this process is the response policy selected by the user.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0187] Conventional visitor response systems only determine the visitor's purpose and notify the resident, but they have the drawback of not being able to adjust the response considering the resident's emotional state. Furthermore, because the response is selected based solely on whether the visitor is suspicious or not, it is difficult to alleviate the psychological stress on the resident. This can lead to situations where residents feel anxious while responding or give inappropriate responses.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes voice acquisition means for acquiring the visitor's voice, data conversion means for converting the voice into text data, data analysis means for analyzing the text data to determine the visitor's requirements, and response adjustment means for adjusting the response action to the visitor based on emotion recognition. This makes it possible to provide an optimal response that takes into account the emotional state of the resident in addition to the visitor's requirements, thereby reducing the resident's psychological stress.

[0190] "Voice acquisition means" refers to a device or method for collecting voice signals and supplying them to a system as input data.

[0191] "Data conversion means" refers to a technology or device that performs the process of converting collected audio signals into text data.

[0192] "Data analysis means" refers to a mechanism or method that performs analysis based on text data to determine the requests and intentions of visitors.

[0193] "Information notification means" refers to a device or method for communicating to the user the content determined based on the analysis results.

[0194] "Emotion recognition means" refers to technology or devices for recognizing a user's emotional state in real time from sources such as voice and facial expressions.

[0195] A "response adjustment mechanism" is a method or mechanism for adjusting the content of the response to a visitor based on the perceived emotional state.

[0196] A "means of decision-making" is a guide or tool that helps a user make the optimal choice based on the information or situation presented.

[0197] "Dialogue support means" refers to technologies or devices that support smooth communication between users and visitors.

[0198] "Response control means" refers to a mechanism or means for playing back a pre-set response for a visitor, or for changing a response based on emotion recognition.

[0199] The system for realizing this invention mainly consists of three main components: a server, a terminal, and a user. The server collects audio data using voice acquisition means for recording the visitor's voice. Subsequently, data conversion means convert the acquired audio data into text format. Based on this text data, the server uses data analysis means to determine the visitor's requests and intentions.

[0200] The terminal can monitor the user's emotional state in real time through emotion recognition means. This involves advanced analytical techniques, such as analyzing voice tone and facial expressions. Based on the identified emotional state, the server adjusts its response to the visitor appropriately via response adjustment means. This allows for optimal support of the visitor without causing stress to the user.

[0201] As a practical application, if a suspicious person visits, the server can automatically reject the response using pre-configured responses. On the other hand, when a familiar delivery person visits, the terminal can recommend allowing direct interaction based on the emotion recognition results, demonstrating flexible responses tailored to the situation.

[0202] An example of a prompt would be: "We are currently developing an app that uses an emotion engine on your home's intercom to determine the visitor's intentions and suggest an appropriate response based on the resident's emotions. Please create a refined text description of this situation."

[0203] The hardware used includes smartphones and network-connected devices, while the software utilizes high-performance APIs for speech recognition. Furthermore, advanced computational processing, such as machine learning models, is required for sentiment analysis, and these processes enable the maintenance of sufficient response accuracy.

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The server records the visitor's voice using voice acquisition equipment. The input is the visitor's voice, and the output is the digital data of that voice. In the voice acquisition process, a capture device such as a microphone is used, and the voice data is converted to an appropriate file format and saved.

[0207] Step 2:

[0208] The server converts digitized audio data into text format using a data conversion mechanism. The input is digital audio data, and the output is corresponding text data. Based on speech recognition technology, a process is performed to analyze each phoneme and convert it into characters.

[0209] Step 3:

[0210] The server analyzes text data to determine the visitor's requests and intentions using data analysis tools. The input is the converted text data, and the output is the analysis result regarding the visitor's intentions. Natural language processing techniques are used to extract important keywords and context to identify the content of the request.

[0211] Step 4:

[0212] The device detects the user's emotional state using emotion recognition technology. The input is the user's voice or video, and the output is the emotional data determined from it. An algorithm operates to analyze voice tone and facial expressions to infer the emotional state.

[0213] Step 5:

[0214] The server determines how to respond to the visitor using response adjustment mechanisms based on the acquired emotional data and analysis results. The input is the output from steps 3 and 4, and the output is the optimized response content for the visitor. This allows the system to automatically select the appropriate response action for the visitor according to their emotional state.

[0215] Step 6:

[0216] The user reviews the response displayed on the terminal and chooses whether or not to respond using the selection tool. The system presents response options as input, and the user's selection is the output. The options include direct response, refusal of response, and postponement of the response time.

[0217] Step 7:

[0218] The terminal manages interactions with visitors using dialogue support tools via a server, based on the user's selection. Input is the result of the user's selection, and output is the result of communication with the visitor. The response to the visitor is executed appropriately according to the selected response policy.

[0219] Step 8:

[0220] The server, if it determines that a visitor is suspicious, plays a pre-configured response via a response control mechanism. The input is the result of the suspicious person detection, and the output is the execution of the automated response. A safe response to the visitor is automatically played.

[0221] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0222] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0223] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0224] [Second Embodiment]

[0225] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0226] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0227] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0228] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0229] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0230] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0231] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0232] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0233] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0234] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0235] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0236] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0237] This invention is an intercom system for automating visitor reception. The system primarily relies on a process of acquiring and analyzing visitor voices to determine the necessary response. Its operation is described below.

[0238] System Overview

[0239] The system is mainly divided into three components: server, terminal, and user. The server primarily handles data processing, the terminal provides the user interface, and the user makes decisions on how to respond based on the system's presentation.

[0240] Program processing

[0241] The server receives the visitor's voice. The system's intercom acquires the visitor's voice in real time and sends that data to the server.

[0242] The server converts speech to text. Using speech recognition technology, it converts the received audio data into text, preparing it for analysis in the next step.

[0243] The server analyzes the text data. The analysis engine evaluates keywords and context within the text and performs filtering to determine whether the visitor's requirements are safe or suspicious.

[0244] The server generates a notification based on the analysis results. Based on the analysis results, it sends a notification to the terminal, informing the user of the visitor's requirements.

[0245] The device displays notifications to the user. For example, the visitor's requirements are displayed on the resident's smartphone or a dedicated terminal inside the room, and audio alerts can also be set up.

[0246] The user chooses to respond based on the notification content. The user can choose to allow or deny the response through their device.

[0247] As an example of this system, if a delivery person arrives to deliver a package, the server recognizes the keyword "delivery" and determines that the visit is safe. As a result, the terminal notifies the user of the package's arrival, and the user can optionally decide whether or not to answer the door. This process protects the user from unnecessary visits and potential dangers. Furthermore, the system records visit history, which can be used to improve the accuracy of decisions regarding subsequent visits.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The server receives the visitor's voice from the intercom microphone. The voice data is converted to a digital format and stored in a form that can be processed internally by the server.

[0251] Step 2:

[0252] The server sends the audio data to the speech recognition engine, which converts it into text data. The speech recognition engine analyzes the content of the audio and outputs it as a string of characters.

[0253] Step 3:

[0254] The server passes the converted text data to the analysis engine, which performs keyword detection and intent evaluation. This helps determine whether the visitor's requirements are safe.

[0255] Step 4:

[0256] The server generates a visitor safety assessment based on the analysis results. If suspicious content is detected, a flag is set to identify the relevant processing result, and a notification preparation state is established.

[0257] Step 5:

[0258] The server transmits the visitor's requirements and analysis results to the terminal. The terminal then informs the user of the situation by displaying the relevant information.

[0259] Step 6:

[0260] The device notifies the user, informing them of the visit via voice alerts or pop-up messages. The user is then given the option to choose whether or not to respond.

[0261] Step 7:

[0262] The user chooses to allow or deny the interaction via their device. If they choose to allow, they are able to communicate directly with the visitor through the intercom.

[0263] Step 8:

[0264] The terminal feeds back the response result to the server based on the user's selection. If the request is rejected, the server plays a pre-configured response message to the visitor.

[0265] Step 9:

[0266] The server logs visit details and user interaction results in a database, which is used to improve the accuracy of future analysis and decision-making.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] Current intercom systems sometimes struggle to accurately identify visitors and provide residents with timely and appropriate information. Furthermore, measures against suspicious individuals and management of visitor history are insufficient. This creates unnecessary inconvenience and exposes residents to potential dangers, thus creating a need for a system that addresses these issues.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes acquisition means for acquiring the visitor's voice, conversion means for converting the voice into text data, analysis means for analyzing the text data to determine the visitor's requirements, notification means for notifying the resident of the requirements, and means for recording the history. This makes it possible to quickly provide visitor information to the resident and manage the visit history.

[0272] "Acquisition means" refers to a device or method that has the function of capturing audio from visitors in real time and transmitting it to a server as digital data.

[0273] "Conversion means" refers to a process that includes speech recognition technology used to convert acquired audio data into an analyzable text format.

[0274] "Analysis means" refers to a system or procedure that uses natural language processing technology to extract the intentions and requirements of visitors from text data and evaluate their content.

[0275] "Notification means" refers to a method and apparatus for appropriately presenting analyzed visitor information to residents and generating and transmitting messages to encourage necessary actions.

[0276] "Selection method" refers to an interface and process that allows residents to intuitively choose response options based on the visitor information presented.

[0277] "Means" is a comprehensive term that refers to the devices and methods used to achieve a specific function within a system.

[0278] As an embodiment of this invention, an intercom system for automating visitor reception will be described. This system mainly consists of three components: a server, a terminal, and a user.

[0279] Servers are the core of data processing. Computer servers are used as hardware to receive visitor voice data in real time and process that information. To convert the voice into digital text, speech recognition software such as Google Speech-to-Text API or Azure Speech Service is used. This converted text data is then analyzed by an analysis engine, which uses natural language processing techniques to analyze keywords and context, revealing the visitor's requirements and intentions. Based on the analysis results, a safety assessment is made.

[0280] The terminal serves as an interface for the user. Smartphones or dedicated indoor terminals are used to provide visitors' information to residents. Notifications are not only displayed on the screen but also transmitted to the user via sound or vibration, enabling quick and appropriate responses to be executed instantly.

[0281] Based on the information provided by the terminal, the user selects a response to the visitor. As a specific example, in the case of a home delivery visit, a user who checks the "parcel arrival" notification can decide whether to immediately permit the response on the spot. This system also has a function to record the visit history, which contributes to further accuracy improvement by analyzing visitor information later.

[0282] An example of a prompt sentence is "Please develop an AI model that explains how to evaluate the safety of a visit and generate appropriate notifications for the user when the visitor says 'It's a home delivery'." In this way, by using the generative AI model, it is possible to build more advanced response logic.

[0283] The flow of the specific process in Example 1 will be described using FIG. 11.

[0284] Step 1:

[0285] The server acquires the voice of the visitor. The voice data received in real time by the intercom is transmitted to the server. This input voice data is stored in the server's storage and serves as material for conversion processing.

[0286] Step 2:

[0287] The server converts the voice data into text data. Using a speech recognition engine, the server analyzes the voice data and generates corresponding text data. In this conversion process, the speech recognition model analyzes the voice pattern and converts it into text as a language. The output is data in which the content of the visitor's speech is texturized.

[0288] Step 3:

[0289] The server analyzes the text data. Using natural language processing techniques, the server analyzes the meaning of the text. This process involves keyword extraction and contextual analysis to identify the visitor's intent. The output includes visitor requirements and a safety assessment.

[0290] Step 4:

[0291] The server generates a notification based on the analysis results. Based on the analyzed data, the server creates a notification message and sends it to the device. This notification includes visitor requirements and recommended actions regarding the response. The output is the notification message to the user.

[0292] Step 5:

[0293] The terminal displays notifications to the user. The interface on the terminal visually and audibly presents notifications received from the server to the user. This allows the user to check visitor information and take necessary actions immediately. The input is the notification message from the server, and the output is the interface display presented to the user.

[0294] Step 6:

[0295] The user selects a response based on the notification. Based on the information displayed on the device, the user decides how to respond to the visitor. In this step, the user chooses whether to allow or deny the response from the provided options, and this selection is immediately reflected in the system. The input is the content of the notification the user saw on the device, and the output is the response the user selected.

[0296] (Application Example 1)

[0297] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0298] Dealing with visitors in a residential setting can increase security risks and complexity due to the inclusion of external factors that may infringe upon residents' lives. In particular, difficulties in dealing with suspicious individuals can threaten residents' safety. Therefore, a system is needed that can quickly and accurately assess the safety of visitors and notify residents, enabling appropriate responses.

[0299] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0300] In this invention, the server includes acquisition means for acquiring the visitor's voice and image, conversion means for converting the voice into text data, and analysis means for analyzing the text data and image data to determine the visitor's requirements and safety. This makes it possible to quickly and accurately determine the visitor's safety and notify the resident of the results.

[0301] "Acquisition means" refers to a device or method for collecting the audio and images of visitors.

[0302] "Conversion means" refers to a technique or method for converting acquired audio data into text data.

[0303] "Analysis means" refers to a process or system that analyzes audio and image data to evaluate the visitor's requirements and safety.

[0304] "Control means" refers to a method or device for notifying residents based on analysis results and for coordinating and managing communication with visitors.

[0305] A "means of choice" is a method or device that provides a function allowing residents to choose how they respond to visitors.

[0306] The "response function" is a feature that automatically initiates communication with visitors deemed safe and informs residents of the situation.

[0307] To implement this invention, first, as an acquisition means for acquiring the voice and image of the visitor, a terminal equipped with a camera and a microphone is used. The voice data acquired by the terminal is transmitted to the server, where it is converted into text data by the conversion means. For example, voice recognition technologies such as Google Cloud Speech-to-Text API can be utilized.

[0308] The server analyzes the converted text data and image data by the analysis means. For the analysis, technologies such as keyword extraction and context analysis are used, and the requirements and safety of the visitor are evaluated. By leveraging cloud-based analysis services such as AWS Lambda, efficient data processing is realized.

[0309] Based on the analysis result, when the control means determines that the visitor is safe, it sends a notification to the resident's smartphone. This notification is utilized as information when the resident selects a response to the visitor. Through the selection function, the resident can decide whether to permit or reject the response and indicate their intention through the terminal.

[0310] As a specific example, it is possible for the resident to directly start communication with a visitor who has been determined to be safe. Examples of prompt texts include content such as "A service that uses the visitor response system to analyze voice data and evaluate the safety of visitors is provided. Please teach me how to convert voice to text using the Google Cloud Speech-to-Text API." This system makes it easier to handle suspicious persons and can improve the security and convenience of the resident.

[0311] The flow of the specific process in Application Example 1 will be described with reference to FIG. 12.

[0312] Step 1:

[0313] The terminal acquires the voice and image of the visitor.

[0314] The input is the visitor's real-time audio and video, which is collected by the microphone and camera built into the terminal. The output includes means for generating the acquired audio and image data and preparing it for transmission to the server. Specifically, the terminal detects the visitor's movements and speech and automatically starts recording video and audio.

[0315] Step 2:

[0316] The server converts the audio data into text data.

[0317] The system receives audio data sent from a terminal as input and generates text data as output. This conversion utilizes a speech recognition API (e.g., Google Cloud Speech-to-Text) to perform data calculations that extract string information from the audio waveform. Specifically, the system temporarily stores the audio data on a server and then calls the API to request recognition processing.

[0318] Step 3:

[0319] The server analyzes the text and image data.

[0320] The input consists of text data obtained through speech conversion and initial image data, which are used to evaluate the visitor's requirements and safety. Keyword extraction and machine learning algorithms are applied to the analysis to detect suspicious individuals and estimate the purpose of their visit. Specifically, the analysis engine on the server parses the text and applies image recognition technology to perform facial recognition.

[0321] Step 4:

[0322] Based on the analysis results, the server sends a notification to the terminal via the control system.

[0323] The system takes the analysis results (e.g., whether the visitor is safe) as input and prepares a notification to be output to the resident's device based on that. The notification includes visitor information and the safety assessment result. Specifically, the server constructs the notification message and sends it to the device as a push notification.

[0324] Step 5:

[0325] The user can choose whether or not to respond to visitors through their device.

[0326] The system takes notifications from the device as input and outputs a choice to allow or deny the interaction. Specifically, the device's user interface presents the resident with options, and the user makes a decision through buttons on the screen. This process also allows the user to warn about visitors deemed unsafe.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] This invention is an intercom system that combines an emotion engine with a visitor reception system to enable responses that take into account the emotional state of the resident. It is mainly performed through three components: a server, a terminal, and a user.

[0329] System Overview

[0330] This system is based on a process of acquiring and analyzing visitors' voices. Furthermore, it can recognize the user's emotional state in real time and adjust its response to visitors based on that information.

[0331] Program processing

[0332] The server acquires the visitor's voice through the intercom. This voice data is then converted into text data using speech recognition technology and analyzed.

[0333] The server analyzes the text data and evaluates the visitor's requirements. It determines whether the visitor is safe or suspicious and passes the result on to the next processing step.

[0334] The device acquires the resident's emotions using an emotion engine. The emotion engine analyzes features such as voice tone, facial expressions, and volume to understand the emotional state in real time.

[0335] The server adjusts response actions based on data from the emotion engine. If a resident is stressed, it enhances automated response messages and softens visitor interactions to reduce stress.

[0336] The terminal presents residents with pre-configured responses and recommends the most appropriate way to respond to visitors. This includes options such as allowing, postponing, or declining the response.

[0337] Specific example

[0338] For example, if a visitor is found to be a salesperson, the server can analyze that information in real time, and if the emotion engine detects anxiety or stress in the resident, it can be configured to automatically refuse service. In the case of a delivery person, if the emotion engine recognizes that the resident is relaxed, a recommendation to allow direct service will be displayed on the terminal.

[0339] This invention allows residents to interact with visitors with psychological peace of mind, reducing daily stress and preventing harm from suspicious individuals.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The server acquires the visitor's voice from the intercom microphone. This audio data is converted to a digital format and stored on the server.

[0343] Step 2:

[0344] The server sends the acquired audio data to the speech recognition engine, which converts the audio into text data. The text data is then used in the subsequent analysis process.

[0345] Step 3:

[0346] The server analyzes text data to determine the visitor's requirements. It detects specific keywords and phrases and performs safety and suspicion assessments.

[0347] Step 4:

[0348] The device detects the user's emotional state using an emotion engine. The emotion engine analyzes the user's voice tone, facial expressions, and movements to evaluate their current emotions in real time.

[0349] Step 5:

[0350] The server receives the evaluation results from the emotion engine and decides on a response action based on the analysis results. For example, if the user is in a stressed state, an automated response message is prepared, and the server is ready to respond flexibly to the visitor.

[0351] Step 6:

[0352] The device displays the visitor's requirements and suggested responses to the user. The user is then given the option to accept, postpone, or decline the interaction.

[0353] Step 7:

[0354] The device will respond to the intercom based on the user's selected response method. If the user allows the call, conversation mode will be enabled; if the user chooses not to answer, a pre-set message will be played.

[0355] Step 8:

[0356] The server records visit details and response results in a database. This record is used for future visit analysis and to improve the accuracy of sentiment recognition.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] The present invention aims to reduce the psychological burden on residents when dealing with visitors and to enable safer and smoother visitor interactions. It also aims to prevent harm from suspicious individuals while enabling flexible responses that take into account the emotional state of residents.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes: an audio acquisition means for acquiring the voice of a visitor; an audio conversion means for converting the voice into text data; a content evaluation means for analyzing the text data to determine the visitor's requirements; an emotion acquisition means for recognizing the emotions of a resident; a response adjustment means for adjusting the response action based on the resident's emotional state acquired by the emotion acquisition means; and a selection presentation means for presenting the adjusted response content to the resident and enabling them to choose how to respond to the visitor. This makes it possible to adjust responses considering the resident's emotional state and to take appropriate measures to prevent harm from suspicious individuals.

[0362] "Voice acquisition means" refers to a device or process for recognizing and recording voices emitted by visitors.

[0363] "Acoustic conversion means" refers to a device or process for analyzing acquired audio data and converting it into a corresponding text data format.

[0364] A "content evaluation tool" is a device or process for analyzing converted text data to determine what the visitor wants.

[0365] An "emotion acquisition method" is a device or process for recognizing emotions by analyzing elements such as voice and facial expressions in order to evaluate the psychological state of a resident.

[0366] A "response adjustment mechanism" is a device or process that adjusts the method of responding to visitors and the messages they send based on the perceived emotional state of the residents.

[0367] A "selection presentation means" is a device or process that presents residents with pre-arranged response options and provides them with multiple choices on how to respond to visitors.

[0368] This invention is a system for handling visitors, and is primarily composed of three components: a server, a terminal, and a user. Its main objective is to acquire and analyze visitors' voices, enabling flexible responses that take into account the emotional state of the resident. By combining voice recognition technology and an emotion recognition engine, it achieves appropriate responses to visitors.

[0369] The server collects visitor voices from the intercom as a means of acquiring audio. The collected audio data is converted into text data using commercially available speech recognition software as a means of audio conversion. This process makes it possible for the user to visualize the visitor's intentions and requests.

[0370] The server then uses the converted text data to employ natural language processing techniques as a content evaluation tool, analyzing the visitor's requirements and intentions. This allows it to determine whether the visitor is safe or suspicious.

[0371] The device provides an emotion acquisition method that uses emotion recognition software to evaluate the emotions of residents. This allows for real-time analysis of residents' voice tone, facial expressions, volume, etc., enabling an understanding of their psychological state.

[0372] The server also includes a response adjustment mechanism that adjusts response actions according to the resident's emotional state, and recommends appropriate responses to provide flexible support if the resident is experiencing stress.

[0373] The system provides a selection tool via a terminal that displays adjusted response options to the user, allowing the user to choose a response policy for the visitor based on this information. These options include allowing, postponing, or not responding.

[0374] As a concrete example, consider a case where a visitor says, "Hello, delivery person." If the emotion acquisition system determines that the user is relaxed, a recommendation to "allow interaction" will be displayed on the device. An example of the prompt text used in this case is as follows: "The system analyzes the visitor's voice and suggests an appropriate response method based on the resident's emotional state. If the visitor's voice says 'Hello, delivery person,' the system will allow interaction with the visitor when the resident is relaxed."

[0375] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0376] Step 1:

[0377] The server acquires the visitor's voice via the intercom. This acquired voice data is used as input. Specifically, the server receives the voice signal from the intercom's microphone in real time and stores it in digital format. The output at this stage is the digitized voice data.

[0378] Step 2:

[0379] The server uses speech recognition technology to convert this audio data into text data. The input is the previously acquired digital audio data. Specifically, the server uses speech recognition software to analyze the audio waveform and convert it into corresponding text. The output of this process is text data representing what the visitor said.

[0380] Step 3:

[0381] The server analyzes the converted text data to determine the visitor's requirements. The input is text data. Specifically, the server applies natural language processing techniques to analyze the content of the text data, understand its meaning, and identify what the visitor is looking for (e.g., delivery person, salesperson). The output of this process is an evaluation result indicating the visitor's requirements.

[0382] Step 4:

[0383] The device acquires the resident's emotions using emotion recognition technology. The input is data such as the resident's voice and facial expressions. Specifically, the device uses an emotion recognition engine to analyze the resident's voice tone and facial expressions to determine their emotional state, such as whether they are relaxed or stressed. The output of this process is data representing the resident's emotional state.

[0384] Step 5:

[0385] The server adjusts its response actions to visitors based on the resident's emotional state. Inputs include resident emotional state data and visitor requirements assessment results. Specifically, the server customizes response messages and methods, adjusting the response to mitigate stress if the resident is experiencing it. The output of this process is the adjusted response.

[0386] Step 6:

[0387] The terminal presents the user with pre-configured response options, allowing them to select the most appropriate response. The input is the pre-configured response content. Specifically, the terminal presents the resident with options such as "Allow response," "Postpone response," and "Do not respond." The user can then choose the most appropriate option. The output of this process is the response policy selected by the user.

[0388] (Application Example 2)

[0389] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0390] Conventional visitor response systems only determine the visitor's purpose and notify the resident, but they have the drawback of not being able to adjust the response considering the resident's emotional state. Furthermore, because the response is selected based solely on whether the visitor is suspicious or not, it is difficult to alleviate the psychological stress on the resident. This can lead to situations where residents feel anxious while responding or give inappropriate responses.

[0391] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0392] In this invention, the server includes voice acquisition means for acquiring the visitor's voice, data conversion means for converting the voice into text data, data analysis means for analyzing the text data to determine the visitor's requirements, and response adjustment means for adjusting the response action to the visitor based on emotion recognition. This makes it possible to provide an optimal response that takes into account the emotional state of the resident in addition to the visitor's requirements, thereby reducing the resident's psychological stress.

[0393] "Voice acquisition means" refers to a device or method for collecting voice signals and supplying them to a system as input data.

[0394] "Data conversion means" refers to a technology or device that performs the process of converting collected audio signals into text data.

[0395] "Data analysis means" refers to a mechanism or method that performs analysis based on text data to determine the requests and intentions of visitors.

[0396] "Information notification means" refers to a device or method for communicating to the user the content determined based on the analysis results.

[0397] "Emotion recognition means" refers to technology or devices for recognizing a user's emotional state in real time from sources such as voice and facial expressions.

[0398] A "response adjustment mechanism" is a method or mechanism for adjusting the content of the response to a visitor based on the perceived emotional state.

[0399] A "means of decision-making" is a guide or tool that helps a user make the optimal choice based on the information or situation presented.

[0400] "Dialogue support means" refers to technologies or devices that support smooth communication between users and visitors.

[0401] "Response control means" refers to a mechanism or means for playing back a pre-set response for a visitor, or for changing a response based on emotion recognition.

[0402] The system for realizing this invention mainly consists of three main components: a server, a terminal, and a user. The server collects audio data using voice acquisition means for recording the visitor's voice. Subsequently, data conversion means convert the acquired audio data into text format. Based on this text data, the server uses data analysis means to determine the visitor's requests and intentions.

[0403] The terminal can monitor the user's emotional state in real time through emotion recognition means. This involves advanced analytical techniques, such as analyzing voice tone and facial expressions. Based on the identified emotional state, the server adjusts its response to the visitor appropriately via response adjustment means. This allows for optimal support of the visitor without causing stress to the user.

[0404] As a practical application, if a suspicious person visits, the server can automatically reject the response using pre-configured responses. On the other hand, when a familiar delivery person visits, the terminal can recommend allowing direct interaction based on the emotion recognition results, demonstrating flexible responses tailored to the situation.

[0405] An example of a prompt would be: "We are currently developing an app that uses an emotion engine on your home's intercom to determine the visitor's intentions and suggest an appropriate response based on the resident's emotions. Please create a refined text description of this situation."

[0406] The hardware used includes smartphones and network-connected devices, while the software utilizes high-performance APIs for speech recognition. Furthermore, advanced computational processing, such as machine learning models, is required for sentiment analysis, and these processes enable the maintenance of sufficient response accuracy.

[0407] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0408] Step 1:

[0409] The server records the visitor's voice using voice acquisition equipment. The input is the visitor's voice, and the output is the digital data of that voice. In the voice acquisition process, a capture device such as a microphone is used, and the voice data is converted to an appropriate file format and saved.

[0410] Step 2:

[0411] The server converts digitized audio data into text format using a data conversion mechanism. The input is digital audio data, and the output is corresponding text data. Based on speech recognition technology, a process is performed to analyze each phoneme and convert it into characters.

[0412] Step 3:

[0413] The server analyzes text data to determine the visitor's requests and intentions using data analysis tools. The input is the converted text data, and the output is the analysis result regarding the visitor's intentions. Natural language processing techniques are used to extract important keywords and context to identify the content of the request.

[0414] Step 4:

[0415] The device detects the user's emotional state using emotion recognition technology. The input is the user's voice or video, and the output is the emotional data determined from it. An algorithm operates to analyze voice tone and facial expressions to infer the emotional state.

[0416] Step 5:

[0417] The server determines how to respond to the visitor using response adjustment mechanisms based on the acquired emotional data and analysis results. The input is the output from steps 3 and 4, and the output is the optimized response content for the visitor. This allows the system to automatically select the appropriate response action for the visitor according to their emotional state.

[0418] Step 6:

[0419] The user reviews the response displayed on the terminal and chooses whether or not to respond using the selection tool. The system presents response options as input, and the user's selection is the output. The options include direct response, refusal of response, and postponement of the response time.

[0420] Step 7:

[0421] The terminal manages interactions with visitors using dialogue support tools via a server, based on the user's selection. Input is the result of the user's selection, and output is the result of communication with the visitor. The response to the visitor is executed appropriately according to the selected response policy.

[0422] Step 8:

[0423] The server, if it determines that a visitor is suspicious, plays a pre-configured response via a response control mechanism. The input is the result of the suspicious person detection, and the output is the execution of the automated response. A safe response to the visitor is automatically played.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0425] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0427] [Third Embodiment]

[0428] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0429] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0430] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0432] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0434] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0435] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0436] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0439] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0440] This invention is an intercom system for automating visitor reception. The system primarily relies on a process of acquiring and analyzing visitor voices to determine the necessary response. Its operation is described below.

[0441] System Overview

[0442] The system is mainly divided into three components: server, terminal, and user. The server primarily handles data processing, the terminal provides the user interface, and the user makes decisions on how to respond based on the system's presentation.

[0443] Program processing

[0444] The server receives the visitor's voice. The system's intercom acquires the visitor's voice in real time and sends that data to the server.

[0445] The server converts speech to text. Using speech recognition technology, it converts the received audio data into text, preparing it for analysis in the next step.

[0446] The server analyzes the text data. The analysis engine evaluates keywords and context within the text and performs filtering to determine whether the visitor's requirements are safe or suspicious.

[0447] The server generates a notification based on the analysis results. Based on the analysis results, it sends a notification to the terminal, informing the user of the visitor's requirements.

[0448] The device displays notifications to the user. For example, the visitor's requirements are displayed on the resident's smartphone or a dedicated terminal inside the room, and audio alerts can also be set up.

[0449] The user chooses to respond based on the notification content. The user can choose to allow or deny the response through their device.

[0450] As an example of this system, if a delivery person arrives to deliver a package, the server recognizes the keyword "delivery" and determines that the visit is safe. As a result, the terminal notifies the user of the package's arrival, and the user can optionally decide whether or not to answer the door. This process protects the user from unnecessary visits and potential dangers. Furthermore, the system records visit history, which can be used to improve the accuracy of decisions regarding subsequent visits.

[0451] The following describes the processing flow.

[0452] Step 1:

[0453] The server receives the visitor's voice from the intercom microphone. The voice data is converted to a digital format and stored in a form that can be processed internally by the server.

[0454] Step 2:

[0455] The server sends the audio data to the speech recognition engine, which converts it into text data. The speech recognition engine analyzes the content of the audio and outputs it as a string of characters.

[0456] Step 3:

[0457] The server passes the converted text data to the analysis engine, which performs keyword detection and intent evaluation. This helps determine whether the visitor's requirements are safe.

[0458] Step 4:

[0459] The server generates a visitor safety assessment based on the analysis results. If suspicious content is detected, a flag is set to identify the relevant processing result, and a notification preparation state is established.

[0460] Step 5:

[0461] The server transmits the visitor's requirements and analysis results to the terminal. The terminal then informs the user of the situation by displaying the relevant information.

[0462] Step 6:

[0463] The device notifies the user, informing them of the visit via voice alerts or pop-up messages. The user is then given the option to choose whether or not to respond.

[0464] Step 7:

[0465] The user chooses to allow or deny the interaction via their device. If they choose to allow, they are able to communicate directly with the visitor through the intercom.

[0466] Step 8:

[0467] The terminal feeds back the response result to the server based on the user's selection. If the request is rejected, the server plays a pre-configured response message to the visitor.

[0468] Step 9:

[0469] The server logs visit details and user interaction results in a database, which is used to improve the accuracy of future analysis and decision-making.

[0470] (Example 1)

[0471] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0472] Current intercom systems sometimes struggle to accurately identify visitors and provide residents with timely and appropriate information. Furthermore, measures against suspicious individuals and management of visitor history are insufficient. This creates unnecessary inconvenience and exposes residents to potential dangers, thus creating a need for a system that addresses these issues.

[0473] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0474] In this invention, the server includes acquisition means for acquiring the visitor's voice, conversion means for converting the voice into text data, analysis means for analyzing the text data to determine the visitor's requirements, notification means for notifying the resident of the requirements, and means for recording the history. This makes it possible to quickly provide visitor information to the resident and manage the visit history.

[0475] "Acquisition means" refers to a device or method that has the function of capturing audio from visitors in real time and transmitting it to a server as digital data.

[0476] "Conversion means" refers to a process that includes speech recognition technology used to convert acquired audio data into an analyzable text format.

[0477] "Analysis means" refers to a system or procedure that uses natural language processing technology to extract the intentions and requirements of visitors from text data and evaluate their content.

[0478] "Notification means" refers to a method and apparatus for appropriately presenting analyzed visitor information to residents and generating and transmitting messages to encourage necessary actions.

[0479] "Selection method" refers to an interface and process that allows residents to intuitively choose response options based on the visitor information presented.

[0480] "Means" is a comprehensive term that refers to the devices and methods used to achieve a specific function within a system.

[0481] As an embodiment of this invention, an intercom system for automating visitor reception will be described. This system mainly consists of three components: a server, a terminal, and a user.

[0482] Servers are the core of data processing. Computer servers are used as hardware to receive visitor voice data in real time and process that information. To convert the voice into digital text, speech recognition software such as Google Speech-to-Text API or Azure Speech Service is used. This converted text data is then analyzed by an analysis engine, which uses natural language processing techniques to analyze keywords and context, revealing the visitor's requirements and intentions. Based on the analysis results, a safety assessment is made.

[0483] The terminal serves as an interface for the user. Smartphones and dedicated indoor terminals are used to provide residents with visitor information. Notifications are not only displayed on the screen but also communicated to the user via voice and vibration, enabling quick and appropriate responses in an instant.

[0484] Users choose how to respond to visitors based on the information provided by their device. For example, in the case of a delivery service visit, a user who has received a notification that "package has arrived" can immediately decide whether to allow the visitor to enter the room. This system also has a function to record visit history, and analyzing visitor information later contributes to further improving accuracy.

[0485] An example of a prompt would be, "Develop an AI model that explains how to assess the safety of a visit and generate an appropriate notification for the user when a visitor says, 'It's a delivery.'" In this way, generative AI models can be used to build more sophisticated response logic.

[0486] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0487] Step 1:

[0488] The server acquires the visitor's voice. The intercom sends the voice data received in real time to the server. This input voice data is stored in the server's storage and used as material for conversion processing.

[0489] Step 2:

[0490] The server converts audio data into text data. Using a speech recognition engine, the server analyzes the audio data and generates corresponding text data. In this conversion process, the speech recognition model analyzes the audio patterns and converts them into text as language. The output is the visitor's spoken content in text format.

[0491] Step 3:

[0492] The server analyzes the text data. Using natural language processing techniques, the server analyzes the meaning of the text. This process involves keyword extraction and contextual analysis to identify the visitor's intent. The output includes visitor requirements and a safety assessment.

[0493] Step 4:

[0494] The server generates a notification based on the analysis results. Based on the analyzed data, the server creates a notification message and sends it to the device. This notification includes visitor requirements and recommended actions regarding the response. The output is the notification message to the user.

[0495] Step 5:

[0496] The terminal displays notifications to the user. The interface on the terminal visually and audibly presents notifications received from the server to the user. This allows the user to check visitor information and take necessary actions immediately. The input is the notification message from the server, and the output is the interface display presented to the user.

[0497] Step 6:

[0498] The user selects a response based on the notification. Based on the information displayed on the device, the user decides how to respond to the visitor. In this step, the user chooses whether to allow or deny the response from the provided options, and this selection is immediately reflected in the system. The input is the content of the notification the user saw on the device, and the output is the response the user selected.

[0499] (Application Example 1)

[0500] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0501] Dealing with visitors in a residential setting can increase security risks and complexity due to the inclusion of external factors that may infringe upon residents' lives. In particular, difficulties in dealing with suspicious individuals can threaten residents' safety. Therefore, a system is needed that can quickly and accurately assess the safety of visitors and notify residents, enabling appropriate responses.

[0502] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0503] In this invention, the server includes acquisition means for acquiring the visitor's voice and image, conversion means for converting the voice into text data, and analysis means for analyzing the text data and image data to determine the visitor's requirements and safety. This makes it possible to quickly and accurately determine the visitor's safety and notify the resident of the results.

[0504] "Acquisition means" refers to a device or method for collecting the audio and images of visitors.

[0505] "Conversion means" refers to a technique or method for converting acquired audio data into text data.

[0506] "Analysis means" refers to a process or system that analyzes audio and image data to evaluate the visitor's requirements and safety.

[0507] "Control means" refers to a method or device for notifying residents based on analysis results and for coordinating and managing communication with visitors.

[0508] A "means of choice" is a method or device that provides a function allowing residents to choose how they respond to visitors.

[0509] The "response function" is a feature that automatically initiates communication with visitors deemed safe and informs residents of the situation.

[0510] To implement this invention, first, a terminal equipped with a camera and microphone is used as an acquisition means for acquiring the visitor's voice and image. The voice data acquired by the terminal is sent to a server, where a conversion means converts the voice into text data. For example, speech recognition technology such as the Google Cloud Speech-to-Text API can be used.

[0511] The server analyzes the converted text and image data using analytical tools. Keyword extraction and contextual analysis techniques are employed to evaluate visitor requirements and security. Efficient data processing is achieved by utilizing cloud-based analytical services such as AWS Lambda.

[0512] Based on the analysis results, if the control system determines that the visitor is safe, it sends a notification to the resident's smartphone. This notification is used as information for the resident to choose how to respond to the visitor. The selection function allows the resident to decide whether to allow or deny the response and express their intention through their device.

[0513] As a concrete example, residents can initiate direct communication with visitors who are deemed safe. An example of a prompt message might be: "We offer a service that uses the visitor response system to analyze voice data and assess the safety of visitors. Please tell me how to convert speech to text using the Google Cloud Speech-to-Text API." This system makes it easier to deal with suspicious individuals and can improve the security and convenience of residents.

[0514] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0515] Step 1:

[0516] The device acquires the visitor's voice and image.

[0517] The input is the visitor's real-time audio and video, which is collected by the microphone and camera built into the terminal. The output includes means for generating the acquired audio and image data and preparing it for transmission to the server. Specifically, the terminal detects the visitor's movements and speech and automatically starts recording video and audio.

[0518] Step 2:

[0519] The server converts the audio data into text data.

[0520] The system receives audio data sent from a terminal as input and generates text data as output. This conversion utilizes a speech recognition API (e.g., Google Cloud Speech-to-Text) to perform data calculations that extract string information from the audio waveform. Specifically, the system temporarily stores the audio data on a server and then calls the API to request recognition processing.

[0521] Step 3:

[0522] The server analyzes the text and image data.

[0523] The input consists of text data obtained through speech conversion and initial image data, which are used to evaluate the visitor's requirements and safety. Keyword extraction and machine learning algorithms are applied to the analysis to detect suspicious individuals and estimate the purpose of their visit. Specifically, the analysis engine on the server parses the text and applies image recognition technology to perform facial recognition.

[0524] Step 4:

[0525] Based on the analysis results, the server sends a notification to the terminal via the control system.

[0526] The system takes the analysis results (e.g., whether the visitor is safe) as input and prepares a notification to be output to the resident's device based on that. The notification includes visitor information and the safety assessment result. Specifically, the server constructs the notification message and sends it to the device as a push notification.

[0527] Step 5:

[0528] The user can choose whether or not to respond to visitors through their device.

[0529] The system takes notifications from the device as input and outputs a choice to allow or deny the interaction. Specifically, the device's user interface presents the resident with options, and the user makes a decision through buttons on the screen. This process also allows the user to warn about visitors deemed unsafe.

[0530] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0531] This invention is an intercom system that combines an emotion engine with a visitor reception system to enable responses that take into account the emotional state of the resident. It is mainly performed through three components: a server, a terminal, and a user.

[0532] System Overview

[0533] This system is based on a process of acquiring and analyzing visitors' voices. Furthermore, it can recognize the user's emotional state in real time and adjust its response to visitors based on that information.

[0534] Program processing

[0535] The server acquires the visitor's voice through the intercom. This voice data is then converted into text data using speech recognition technology and analyzed.

[0536] The server analyzes the text data and evaluates the visitor's requirements. It determines whether the visitor is safe or suspicious and passes the result on to the next processing step.

[0537] The device acquires the resident's emotions using an emotion engine. The emotion engine analyzes features such as voice tone, facial expressions, and volume to understand the emotional state in real time.

[0538] The server adjusts response actions based on data from the emotion engine. If a resident is stressed, it enhances automated response messages and softens visitor interactions to reduce stress.

[0539] The terminal presents residents with pre-configured responses and recommends the most appropriate way to respond to visitors. This includes options such as allowing, postponing, or declining the response.

[0540] Specific example

[0541] For example, if a visitor is found to be a salesperson, the server can analyze that information in real time, and if the emotion engine detects anxiety or stress in the resident, it can be configured to automatically refuse service. In the case of a delivery person, if the emotion engine recognizes that the resident is relaxed, a recommendation to allow direct service will be displayed on the terminal.

[0542] This invention allows residents to interact with visitors with psychological peace of mind, reducing daily stress and preventing harm from suspicious individuals.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The server acquires the visitor's voice from the intercom microphone. This audio data is converted to a digital format and stored on the server.

[0546] Step 2:

[0547] The server sends the acquired audio data to the speech recognition engine, which converts the audio into text data. The text data is then used in the subsequent analysis process.

[0548] Step 3:

[0549] The server analyzes text data to determine the visitor's requirements. It detects specific keywords and phrases and performs safety and suspicion assessments.

[0550] Step 4:

[0551] The device detects the user's emotional state using an emotion engine. The emotion engine analyzes the user's voice tone, facial expressions, and movements to evaluate their current emotions in real time.

[0552] Step 5:

[0553] The server receives the evaluation results from the emotion engine and decides on a response action based on the analysis results. For example, if the user is in a stressed state, an automated response message is prepared, and the server is ready to respond flexibly to the visitor.

[0554] Step 6:

[0555] The device displays the visitor's requirements and suggested responses to the user. The user is then given the option to accept, postpone, or decline the interaction.

[0556] Step 7:

[0557] The device will respond to the intercom based on the user's selected response method. If the user allows the call, conversation mode will be enabled; if the user chooses not to answer, a pre-set message will be played.

[0558] Step 8:

[0559] The server records visit details and response results in a database. This record is used for future visit analysis and to improve the accuracy of sentiment recognition.

[0560] (Example 2)

[0561] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] The present invention aims to reduce the psychological burden on residents when dealing with visitors and to enable safer and smoother visitor interactions. It also aims to prevent harm from suspicious individuals while enabling flexible responses that take into account the emotional state of residents.

[0563] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0564] In this invention, the server includes: an audio acquisition means for acquiring the voice of a visitor; an audio conversion means for converting the voice into text data; a content evaluation means for analyzing the text data to determine the visitor's requirements; an emotion acquisition means for recognizing the emotions of a resident; a response adjustment means for adjusting the response action based on the resident's emotional state acquired by the emotion acquisition means; and a selection presentation means for presenting the adjusted response content to the resident and enabling them to choose how to respond to the visitor. This makes it possible to adjust responses considering the resident's emotional state and to take appropriate measures to prevent harm from suspicious individuals.

[0565] "Voice acquisition means" refers to a device or process for recognizing and recording voices emitted by visitors.

[0566] "Acoustic conversion means" refers to a device or process for analyzing acquired audio data and converting it into a corresponding text data format.

[0567] A "content evaluation tool" is a device or process for analyzing converted text data to determine what the visitor wants.

[0568] An "emotion acquisition method" is a device or process for recognizing emotions by analyzing elements such as voice and facial expressions in order to evaluate the psychological state of a resident.

[0569] A "response adjustment mechanism" is a device or process that adjusts the method of responding to visitors and the messages they send based on the perceived emotional state of the residents.

[0570] A "selection presentation means" is a device or process that presents residents with pre-arranged response options and provides them with multiple choices on how to respond to visitors.

[0571] This invention is a system for handling visitors, and is primarily composed of three components: a server, a terminal, and a user. Its main objective is to acquire and analyze visitors' voices, enabling flexible responses that take into account the emotional state of the resident. By combining voice recognition technology and an emotion recognition engine, it achieves appropriate responses to visitors.

[0572] The server collects visitor voices from the intercom as a means of acquiring audio. The collected audio data is converted into text data using commercially available speech recognition software as a means of audio conversion. This process makes it possible for the user to visualize the visitor's intentions and requests.

[0573] The server then uses the converted text data to employ natural language processing techniques as a content evaluation tool, analyzing the visitor's requirements and intentions. This allows it to determine whether the visitor is safe or suspicious.

[0574] The device provides an emotion acquisition method that uses emotion recognition software to evaluate the emotions of residents. This allows for real-time analysis of residents' voice tone, facial expressions, volume, etc., enabling an understanding of their psychological state.

[0575] The server also includes a response adjustment mechanism that adjusts response actions according to the resident's emotional state, and recommends appropriate responses to provide flexible support if the resident is experiencing stress.

[0576] The system provides a selection tool via a terminal that displays adjusted response options to the user, allowing the user to choose a response policy for the visitor based on this information. These options include allowing, postponing, or not responding.

[0577] As a concrete example, consider a case where a visitor says, "Hello, delivery person." If the emotion acquisition system determines that the user is relaxed, a recommendation to "allow interaction" will be displayed on the device. An example of the prompt text used in this case is as follows: "The system analyzes the visitor's voice and suggests an appropriate response method based on the resident's emotional state. If the visitor's voice says 'Hello, delivery person,' the system will allow interaction with the visitor when the resident is relaxed."

[0578] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0579] Step 1:

[0580] The server acquires the visitor's voice via the intercom. This acquired voice data is used as input. Specifically, the server receives the voice signal from the intercom's microphone in real time and stores it in digital format. The output at this stage is the digitized voice data.

[0581] Step 2:

[0582] The server uses speech recognition technology to convert this audio data into text data. The input is the previously acquired digital audio data. Specifically, the server uses speech recognition software to analyze the audio waveform and convert it into corresponding text. The output of this process is text data representing what the visitor said.

[0583] Step 3:

[0584] The server analyzes the converted text data to determine the visitor's requirements. The input is text data. Specifically, the server applies natural language processing techniques to analyze the content of the text data, understand its meaning, and identify what the visitor is looking for (e.g., delivery person, salesperson). The output of this process is an evaluation result indicating the visitor's requirements.

[0585] Step 4:

[0586] The device acquires the resident's emotions using emotion recognition technology. The input is data such as the resident's voice and facial expressions. Specifically, the device uses an emotion recognition engine to analyze the resident's voice tone and facial expressions to determine their emotional state, such as whether they are relaxed or stressed. The output of this process is data representing the resident's emotional state.

[0587] Step 5:

[0588] The server adjusts its response actions to visitors based on the resident's emotional state. Inputs include resident emotional state data and visitor requirements assessment results. Specifically, the server customizes response messages and methods, adjusting the response to mitigate stress if the resident is experiencing it. The output of this process is the adjusted response.

[0589] Step 6:

[0590] The terminal presents the user with pre-configured response options, allowing them to select the most appropriate response. The input is the pre-configured response content. Specifically, the terminal presents the resident with options such as "Allow response," "Postpone response," and "Do not respond." The user can then choose the most appropriate option. The output of this process is the response policy selected by the user.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] Conventional visitor response systems only determine the visitor's purpose and notify the resident, but they have the drawback of not being able to adjust the response considering the resident's emotional state. Furthermore, because the response is selected based solely on whether the visitor is suspicious or not, it is difficult to alleviate the psychological stress on the resident. This can lead to situations where residents feel anxious while responding or give inappropriate responses.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes voice acquisition means for acquiring the visitor's voice, data conversion means for converting the voice into text data, data analysis means for analyzing the text data to determine the visitor's requirements, and response adjustment means for adjusting the response action to the visitor based on emotion recognition. This makes it possible to provide an optimal response that takes into account the emotional state of the resident in addition to the visitor's requirements, thereby reducing the resident's psychological stress.

[0596] "Voice acquisition means" refers to a device or method for collecting voice signals and supplying them to a system as input data.

[0597] "Data conversion means" refers to a technology or device that performs the process of converting collected audio signals into text data.

[0598] "Data analysis means" refers to a mechanism or method that performs analysis based on text data to determine the requests and intentions of visitors.

[0599] "Information notification means" refers to a device or method for communicating to the user the content determined based on the analysis results.

[0600] "Emotion recognition means" refers to technology or devices for recognizing a user's emotional state in real time from sources such as voice and facial expressions.

[0601] A "response adjustment mechanism" is a method or mechanism for adjusting the content of the response to a visitor based on the perceived emotional state.

[0602] A "means of decision-making" is a guide or tool that helps a user make the optimal choice based on the information or situation presented.

[0603] "Dialogue support means" refers to technologies or devices that support smooth communication between users and visitors.

[0604] "Response control means" refers to a mechanism or means for playing back a pre-set response for a visitor, or for changing a response based on emotion recognition.

[0605] The system for realizing this invention mainly consists of three main components: a server, a terminal, and a user. The server collects audio data using voice acquisition means for recording the visitor's voice. Subsequently, data conversion means convert the acquired audio data into text format. Based on this text data, the server uses data analysis means to determine the visitor's requests and intentions.

[0606] The terminal can monitor the user's emotional state in real time through emotion recognition means. This involves advanced analytical techniques, such as analyzing voice tone and facial expressions. Based on the identified emotional state, the server adjusts its response to the visitor appropriately via response adjustment means. This allows for optimal support of the visitor without causing stress to the user.

[0607] As a practical application, if a suspicious person visits, the server can automatically reject the response using pre-configured responses. On the other hand, when a familiar delivery person visits, the terminal can recommend allowing direct interaction based on the emotion recognition results, demonstrating flexible responses tailored to the situation.

[0608] An example of a prompt would be: "We are currently developing an app that uses an emotion engine on your home's intercom to determine the visitor's intentions and suggest an appropriate response based on the resident's emotions. Please create a refined text description of this situation."

[0609] The hardware used includes smartphones and network-connected devices, while the software utilizes high-performance APIs for speech recognition. Furthermore, advanced computational processing, such as machine learning models, is required for sentiment analysis, and these processes enable the maintenance of sufficient response accuracy.

[0610] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0611] Step 1:

[0612] The server records the visitor's voice using voice acquisition equipment. The input is the visitor's voice, and the output is the digital data of that voice. In the voice acquisition process, a capture device such as a microphone is used, and the voice data is converted to an appropriate file format and saved.

[0613] Step 2:

[0614] The server converts digitized audio data into text format using a data conversion mechanism. The input is digital audio data, and the output is corresponding text data. Based on speech recognition technology, a process is performed to analyze each phoneme and convert it into characters.

[0615] Step 3:

[0616] The server analyzes text data to determine the visitor's requests and intentions using data analysis tools. The input is the converted text data, and the output is the analysis result regarding the visitor's intentions. Natural language processing techniques are used to extract important keywords and context to identify the content of the request.

[0617] Step 4:

[0618] The device detects the user's emotional state using emotion recognition technology. The input is the user's voice or video, and the output is the emotional data determined from it. An algorithm operates to analyze voice tone and facial expressions to infer the emotional state.

[0619] Step 5:

[0620] The server determines how to respond to the visitor using response adjustment mechanisms based on the acquired emotional data and analysis results. The input is the output from steps 3 and 4, and the output is the optimized response content for the visitor. This allows the system to automatically select the appropriate response action for the visitor according to their emotional state.

[0621] Step 6:

[0622] The user reviews the response displayed on the terminal and chooses whether or not to respond using the selection tool. The system presents response options as input, and the user's selection is the output. The options include direct response, refusal of response, and postponement of the response time.

[0623] Step 7:

[0624] The terminal manages interactions with visitors using dialogue support tools via a server, based on the user's selection. Input is the result of the user's selection, and output is the result of communication with the visitor. The response to the visitor is executed appropriately according to the selected response policy.

[0625] Step 8:

[0626] The server, if it determines that a visitor is suspicious, plays a pre-configured response via a response control mechanism. The input is the result of the suspicious person detection, and the output is the execution of the automated response. A safe response to the visitor is automatically played.

[0627] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0628] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0629] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0630] [Fourth Embodiment]

[0631] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0632] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0633] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0634] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0635] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0636] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0637] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0638] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0639] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0640] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0641] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0642] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0643] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] This invention is an intercom system for automating visitor reception. The system primarily relies on a process of acquiring and analyzing visitor voices to determine the necessary response. Its operation is described below.

[0645] System Overview

[0646] The system is mainly divided into three components: server, terminal, and user. The server primarily handles data processing, the terminal provides the user interface, and the user makes decisions on how to respond based on the system's presentation.

[0647] Program processing

[0648] The server receives the visitor's voice. The system's intercom acquires the visitor's voice in real time and sends that data to the server.

[0649] The server converts speech to text. Using speech recognition technology, it converts the received audio data into text, preparing it for analysis in the next step.

[0650] The server analyzes the text data. The analysis engine evaluates keywords and context within the text and performs filtering to determine whether the visitor's requirements are safe or suspicious.

[0651] The server generates a notification based on the analysis results. Based on the analysis results, it sends a notification to the terminal, informing the user of the visitor's requirements.

[0652] The device displays notifications to the user. For example, the visitor's requirements are displayed on the resident's smartphone or a dedicated terminal inside the room, and audio alerts can also be set up.

[0653] The user chooses to respond based on the notification content. The user can choose to allow or deny the response through their device.

[0654] As an example of this system, if a delivery person arrives to deliver a package, the server recognizes the keyword "delivery" and determines that the visit is safe. As a result, the terminal notifies the user of the package's arrival, and the user can optionally decide whether or not to answer the door. This process protects the user from unnecessary visits and potential dangers. Furthermore, the system records visit history, which can be used to improve the accuracy of decisions regarding subsequent visits.

[0655] The following describes the processing flow.

[0656] Step 1:

[0657] The server receives the visitor's voice from the intercom microphone. The voice data is converted to a digital format and stored in a form that can be processed internally by the server.

[0658] Step 2:

[0659] The server sends the audio data to the speech recognition engine, which converts it into text data. The speech recognition engine analyzes the content of the audio and outputs it as a string of characters.

[0660] Step 3:

[0661] The server passes the converted text data to the analysis engine, which performs keyword detection and intent evaluation. This helps determine whether the visitor's requirements are safe.

[0662] Step 4:

[0663] The server generates a visitor safety assessment based on the analysis results. If suspicious content is detected, a flag is set to identify the relevant processing result, and a notification preparation state is established.

[0664] Step 5:

[0665] The server transmits the visitor's requirements and analysis results to the terminal. The terminal then informs the user of the situation by displaying the relevant information.

[0666] Step 6:

[0667] The device notifies the user, informing them of the visit via voice alerts or pop-up messages. The user is then given the option to choose whether or not to respond.

[0668] Step 7:

[0669] The user chooses to allow or deny the interaction via their device. If they choose to allow, they are able to communicate directly with the visitor through the intercom.

[0670] Step 8:

[0671] The terminal feeds back the response result to the server based on the user's selection. If the request is rejected, the server plays a pre-configured response message to the visitor.

[0672] Step 9:

[0673] The server logs visit details and user interaction results in a database, which is used to improve the accuracy of future analysis and decision-making.

[0674] (Example 1)

[0675] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] Current intercom systems sometimes struggle to accurately identify visitors and provide residents with timely and appropriate information. Furthermore, measures against suspicious individuals and management of visitor history are insufficient. This creates unnecessary inconvenience and exposes residents to potential dangers, thus creating a need for a system that addresses these issues.

[0677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0678] In this invention, the server includes acquisition means for acquiring the visitor's voice, conversion means for converting the voice into text data, analysis means for analyzing the text data to determine the visitor's requirements, notification means for notifying the resident of the requirements, and means for recording the history. This makes it possible to quickly provide visitor information to the resident and manage the visit history.

[0679] "Acquisition means" refers to a device or method that has the function of capturing audio from visitors in real time and transmitting it to a server as digital data.

[0680] "Conversion means" refers to a process that includes speech recognition technology used to convert acquired audio data into an analyzable text format.

[0681] "Analysis means" refers to a system or procedure that uses natural language processing technology to extract the intentions and requirements of visitors from text data and evaluate their content.

[0682] "Notification means" refers to a method and apparatus for appropriately presenting analyzed visitor information to residents and generating and transmitting messages to encourage necessary actions.

[0683] "Selection method" refers to an interface and process that allows residents to intuitively choose response options based on the visitor information presented.

[0684] "Means" is a comprehensive term that refers to the devices and methods used to achieve a specific function within a system.

[0685] As an embodiment of this invention, an intercom system for automating visitor reception will be described. This system mainly consists of three components: a server, a terminal, and a user.

[0686] Servers are the core of data processing. Computer servers are used as hardware to receive visitor voice data in real time and process that information. To convert the voice into digital text, speech recognition software such as Google Speech-to-Text API or Azure Speech Service is used. This converted text data is then analyzed by an analysis engine, which uses natural language processing techniques to analyze keywords and context, revealing the visitor's requirements and intentions. Based on the analysis results, a safety assessment is made.

[0687] The terminal serves as an interface for the user. Smartphones and dedicated indoor terminals are used to provide residents with visitor information. Notifications are not only displayed on the screen but also communicated to the user via voice and vibration, enabling quick and appropriate responses in an instant.

[0688] Users choose how to respond to visitors based on the information provided by their device. For example, in the case of a delivery service visit, a user who has received a notification that "package has arrived" can immediately decide whether to allow the visitor to enter the room. This system also has a function to record visit history, and analyzing visitor information later contributes to further improving accuracy.

[0689] An example of a prompt would be, "Develop an AI model that explains how to assess the safety of a visit and generate an appropriate notification for the user when a visitor says, 'It's a delivery.'" In this way, generative AI models can be used to build more sophisticated response logic.

[0690] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0691] Step 1:

[0692] The server acquires the visitor's voice. The intercom sends the voice data received in real time to the server. This input voice data is stored in the server's storage and used as material for conversion processing.

[0693] Step 2:

[0694] The server converts audio data into text data. Using a speech recognition engine, the server analyzes the audio data and generates corresponding text data. In this conversion process, the speech recognition model analyzes the audio patterns and converts them into text as language. The output is the visitor's spoken content in text format.

[0695] Step 3:

[0696] The server analyzes the text data. Using natural language processing techniques, the server analyzes the meaning of the text. This process involves keyword extraction and contextual analysis to identify the visitor's intent. The output includes visitor requirements and a safety assessment.

[0697] Step 4:

[0698] The server generates a notification based on the analysis results. Based on the analyzed data, the server creates a notification message and sends it to the device. This notification includes visitor requirements and recommended actions regarding the response. The output is the notification message to the user.

[0699] Step 5:

[0700] The terminal displays notifications to the user. The interface on the terminal visually and audibly presents notifications received from the server to the user. This allows the user to check visitor information and take necessary actions immediately. The input is the notification message from the server, and the output is the interface display presented to the user.

[0701] Step 6:

[0702] The user selects a response based on the notification. Based on the information displayed on the device, the user decides how to respond to the visitor. In this step, the user chooses whether to allow or deny the response from the provided options, and this selection is immediately reflected in the system. The input is the content of the notification the user saw on the device, and the output is the response the user selected.

[0703] (Application Example 1)

[0704] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0705] Dealing with visitors in a residential setting can increase security risks and complexity due to the inclusion of external factors that may infringe upon residents' lives. In particular, difficulties in dealing with suspicious individuals can threaten residents' safety. Therefore, a system is needed that can quickly and accurately assess the safety of visitors and notify residents, enabling appropriate responses.

[0706] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0707] In this invention, the server includes acquisition means for acquiring the visitor's voice and image, conversion means for converting the voice into text data, and analysis means for analyzing the text data and image data to determine the visitor's requirements and safety. This makes it possible to quickly and accurately determine the visitor's safety and notify the resident of the results.

[0708] "Acquisition means" refers to a device or method for collecting the audio and images of visitors.

[0709] "Conversion means" refers to a technique or method for converting acquired audio data into text data.

[0710] "Analysis means" refers to a process or system that analyzes audio and image data to evaluate the visitor's requirements and safety.

[0711] "Control means" refers to a method or device for notifying residents based on analysis results and for coordinating and managing communication with visitors.

[0712] A "means of choice" is a method or device that provides a function allowing residents to choose how they respond to visitors.

[0713] The "response function" is a feature that automatically initiates communication with visitors deemed safe and informs residents of the situation.

[0714] To implement this invention, first, a terminal equipped with a camera and microphone is used as an acquisition means for acquiring the visitor's voice and image. The voice data acquired by the terminal is sent to a server, where a conversion means converts the voice into text data. For example, speech recognition technology such as the Google Cloud Speech-to-Text API can be used.

[0715] The server analyzes the converted text and image data using analytical tools. Keyword extraction and contextual analysis techniques are employed to evaluate visitor requirements and security. Efficient data processing is achieved by utilizing cloud-based analytical services such as AWS Lambda.

[0716] Based on the analysis results, if the control system determines that the visitor is safe, it sends a notification to the resident's smartphone. This notification is used as information for the resident to choose how to respond to the visitor. The selection function allows the resident to decide whether to allow or deny the response and express their intention through their device.

[0717] As a concrete example, residents can initiate direct communication with visitors who are deemed safe. An example of a prompt message might be: "We offer a service that uses the visitor response system to analyze voice data and assess the safety of visitors. Please tell me how to convert speech to text using the Google Cloud Speech-to-Text API." This system makes it easier to deal with suspicious individuals and can improve the security and convenience of residents.

[0718] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0719] Step 1:

[0720] The device acquires the visitor's voice and image.

[0721] The input is the visitor's real-time audio and video, which is collected by the microphone and camera built into the terminal. The output includes means for generating the acquired audio and image data and preparing it for transmission to the server. Specifically, the terminal detects the visitor's movements and speech and automatically starts recording video and audio.

[0722] Step 2:

[0723] The server converts the audio data into text data.

[0724] The system receives audio data sent from a terminal as input and generates text data as output. This conversion utilizes a speech recognition API (e.g., Google Cloud Speech-to-Text) to perform data calculations that extract string information from the audio waveform. Specifically, the system temporarily stores the audio data on a server and then calls the API to request recognition processing.

[0725] Step 3:

[0726] The server analyzes the text and image data.

[0727] The input consists of text data obtained through speech conversion and initial image data, which are used to evaluate the visitor's requirements and safety. Keyword extraction and machine learning algorithms are applied to the analysis to detect suspicious individuals and estimate the purpose of their visit. Specifically, the analysis engine on the server parses the text and applies image recognition technology to perform facial recognition.

[0728] Step 4:

[0729] Based on the analysis results, the server sends a notification to the terminal via the control system.

[0730] The system takes the analysis results (e.g., whether the visitor is safe) as input and prepares a notification to be output to the resident's device based on that. The notification includes visitor information and the safety assessment result. Specifically, the server constructs the notification message and sends it to the device as a push notification.

[0731] Step 5:

[0732] The user can choose whether or not to respond to visitors through their device.

[0733] The system takes notifications from the device as input and outputs a choice to allow or deny the interaction. Specifically, the device's user interface presents the resident with options, and the user makes a decision through buttons on the screen. This process also allows the user to warn about visitors deemed unsafe.

[0734] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0735] This invention is an intercom system that combines an emotion engine with a visitor reception system to enable responses that take into account the emotional state of the resident. It is mainly performed through three components: a server, a terminal, and a user.

[0736] System Overview

[0737] This system is based on a process of acquiring and analyzing visitors' voices. Furthermore, it can recognize the user's emotional state in real time and adjust its response to visitors based on that information.

[0738] Program processing

[0739] The server acquires the visitor's voice through the intercom. This voice data is then converted into text data using speech recognition technology and analyzed.

[0740] The server analyzes the text data and evaluates the visitor's requirements. It determines whether the visitor is safe or suspicious and passes the result on to the next processing step.

[0741] The device acquires the resident's emotions using an emotion engine. The emotion engine analyzes features such as voice tone, facial expressions, and volume to understand the emotional state in real time.

[0742] The server adjusts response actions based on data from the emotion engine. If a resident is stressed, it enhances automated response messages and softens visitor interactions to reduce stress.

[0743] The terminal presents residents with pre-configured responses and recommends the most appropriate way to respond to visitors. This includes options such as allowing, postponing, or declining the response.

[0744] Specific example

[0745] For example, if a visitor is found to be a salesperson, the server can analyze that information in real time, and if the emotion engine detects anxiety or stress in the resident, it can be configured to automatically refuse service. In the case of a delivery person, if the emotion engine recognizes that the resident is relaxed, a recommendation to allow direct service will be displayed on the terminal.

[0746] This invention allows residents to interact with visitors with psychological peace of mind, reducing daily stress and preventing harm from suspicious individuals.

[0747] The following describes the processing flow.

[0748] Step 1:

[0749] The server acquires the visitor's voice from the intercom microphone. This audio data is converted to a digital format and stored on the server.

[0750] Step 2:

[0751] The server sends the acquired audio data to the speech recognition engine, which converts the audio into text data. The text data is then used in the subsequent analysis process.

[0752] Step 3:

[0753] The server analyzes text data to determine the visitor's requirements. It detects specific keywords and phrases and performs safety and suspicion assessments.

[0754] Step 4:

[0755] The device detects the user's emotional state using an emotion engine. The emotion engine analyzes the user's voice tone, facial expressions, and movements to evaluate their current emotions in real time.

[0756] Step 5:

[0757] The server receives the evaluation results from the emotion engine and decides on a response action based on the analysis results. For example, if the user is in a stressed state, an automated response message is prepared, and the server is ready to respond flexibly to the visitor.

[0758] Step 6:

[0759] The device displays the visitor's requirements and suggested responses to the user. The user is then given the option to accept, postpone, or decline the interaction.

[0760] Step 7:

[0761] The device will respond to the intercom based on the user's selected response method. If the user allows the call, conversation mode will be enabled; if the user chooses not to answer, a pre-set message will be played.

[0762] Step 8:

[0763] The server records visit details and response results in a database. This record is used for future visit analysis and to improve the accuracy of sentiment recognition.

[0764] (Example 2)

[0765] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] The present invention aims to reduce the psychological burden on residents when dealing with visitors and to enable safer and smoother visitor interactions. It also aims to prevent harm from suspicious individuals while enabling flexible responses that take into account the emotional state of residents.

[0767] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0768] In this invention, the server includes: an audio acquisition means for acquiring the voice of a visitor; an audio conversion means for converting the voice into text data; a content evaluation means for analyzing the text data to determine the visitor's requirements; an emotion acquisition means for recognizing the emotions of a resident; a response adjustment means for adjusting the response action based on the resident's emotional state acquired by the emotion acquisition means; and a selection presentation means for presenting the adjusted response content to the resident and enabling them to choose how to respond to the visitor. This makes it possible to adjust responses considering the resident's emotional state and to take appropriate measures to prevent harm from suspicious individuals.

[0769] "Voice acquisition means" refers to a device or process for recognizing and recording voices emitted by visitors.

[0770] "Acoustic conversion means" refers to a device or process for analyzing acquired audio data and converting it into a corresponding text data format.

[0771] A "content evaluation tool" is a device or process for analyzing converted text data to determine what the visitor wants.

[0772] An "emotion acquisition method" is a device or process for recognizing emotions by analyzing elements such as voice and facial expressions in order to evaluate the psychological state of a resident.

[0773] A "response adjustment mechanism" is a device or process that adjusts the method of responding to visitors and the messages they send based on the perceived emotional state of the residents.

[0774] A "selection presentation means" is a device or process that presents residents with pre-arranged response options and provides them with multiple choices on how to respond to visitors.

[0775] This invention is a system for handling visitors, and is primarily composed of three components: a server, a terminal, and a user. Its main objective is to acquire and analyze visitors' voices, enabling flexible responses that take into account the emotional state of the resident. By combining voice recognition technology and an emotion recognition engine, it achieves appropriate responses to visitors.

[0776] The server collects visitor voices from the intercom as a means of acquiring audio. The collected audio data is converted into text data using commercially available speech recognition software as a means of audio conversion. This process makes it possible for the user to visualize the visitor's intentions and requests.

[0777] The server then uses the converted text data to employ natural language processing techniques as a content evaluation tool, analyzing the visitor's requirements and intentions. This allows it to determine whether the visitor is safe or suspicious.

[0778] The device provides an emotion acquisition method that uses emotion recognition software to evaluate the emotions of residents. This allows for real-time analysis of residents' voice tone, facial expressions, volume, etc., enabling an understanding of their psychological state.

[0779] The server also includes a response adjustment mechanism that adjusts response actions according to the resident's emotional state, and recommends appropriate responses to provide flexible support if the resident is experiencing stress.

[0780] The system provides a selection tool via a terminal that displays adjusted response options to the user, allowing the user to choose a response policy for the visitor based on this information. These options include allowing, postponing, or not responding.

[0781] As a concrete example, consider a case where a visitor says, "Hello, delivery person." If the emotion acquisition system determines that the user is relaxed, a recommendation to "allow interaction" will be displayed on the device. An example of the prompt text used in this case is as follows: "The system analyzes the visitor's voice and suggests an appropriate response method based on the resident's emotional state. If the visitor's voice says 'Hello, delivery person,' the system will allow interaction with the visitor when the resident is relaxed."

[0782] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0783] Step 1:

[0784] The server acquires the visitor's voice via the intercom. This acquired voice data is used as input. Specifically, the server receives the voice signal from the intercom's microphone in real time and stores it in digital format. The output at this stage is the digitized voice data.

[0785] Step 2:

[0786] The server uses speech recognition technology to convert this audio data into text data. The input is the previously acquired digital audio data. Specifically, the server uses speech recognition software to analyze the audio waveform and convert it into corresponding text. The output of this process is text data representing what the visitor said.

[0787] Step 3:

[0788] The server analyzes the converted text data to determine the visitor's requirements. The input is text data. Specifically, the server applies natural language processing techniques to analyze the content of the text data, understand its meaning, and identify what the visitor is looking for (e.g., delivery person, salesperson). The output of this process is an evaluation result indicating the visitor's requirements.

[0789] Step 4:

[0790] The device acquires the resident's emotions using emotion recognition technology. The input is data such as the resident's voice and facial expressions. Specifically, the device uses an emotion recognition engine to analyze the resident's voice tone and facial expressions to determine their emotional state, such as whether they are relaxed or stressed. The output of this process is data representing the resident's emotional state.

[0791] Step 5:

[0792] The server adjusts its response actions to visitors based on the resident's emotional state. Inputs include resident emotional state data and visitor requirements assessment results. Specifically, the server customizes response messages and methods, adjusting the response to mitigate stress if the resident is experiencing it. The output of this process is the adjusted response.

[0793] Step 6:

[0794] The terminal presents the user with pre-configured response options, allowing them to select the most appropriate response. The input is the pre-configured response content. Specifically, the terminal presents the resident with options such as "Allow response," "Postpone response," and "Do not respond." The user can then choose the most appropriate option. The output of this process is the response policy selected by the user.

[0795] (Application Example 2)

[0796] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] Conventional visitor response systems only determine the visitor's purpose and notify the resident, but they have the drawback of not being able to adjust the response considering the resident's emotional state. Furthermore, because the response is selected based solely on whether the visitor is suspicious or not, it is difficult to alleviate the psychological stress on the resident. This can lead to situations where residents feel anxious while responding or give inappropriate responses.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0799] In this invention, the server includes voice acquisition means for acquiring the visitor's voice, data conversion means for converting the voice into text data, data analysis means for analyzing the text data to determine the visitor's requirements, and response adjustment means for adjusting the response action to the visitor based on emotion recognition. This makes it possible to provide an optimal response that takes into account the emotional state of the resident in addition to the visitor's requirements, thereby reducing the resident's psychological stress.

[0800] "Voice acquisition means" refers to a device or method for collecting voice signals and supplying them to a system as input data.

[0801] "Data conversion means" refers to a technology or device that performs the process of converting collected audio signals into text data.

[0802] "Data analysis means" refers to a mechanism or method that performs analysis based on text data to determine the requests and intentions of visitors.

[0803] "Information notification means" refers to a device or method for communicating to the user the content determined based on the analysis results.

[0804] "Emotion recognition means" refers to technology or devices for recognizing a user's emotional state in real time from sources such as voice and facial expressions.

[0805] A "response adjustment mechanism" is a method or mechanism for adjusting the content of the response to a visitor based on the perceived emotional state.

[0806] A "means of decision-making" is a guide or tool that helps a user make the optimal choice based on the information or situation presented.

[0807] "Dialogue support means" refers to technologies or devices that support smooth communication between users and visitors.

[0808] "Response control means" refers to a mechanism or means for playing back a pre-set response for a visitor, or for changing a response based on emotion recognition.

[0809] The system for realizing this invention mainly consists of three main components: a server, a terminal, and a user. The server collects audio data using voice acquisition means for recording the visitor's voice. Subsequently, data conversion means convert the acquired audio data into text format. Based on this text data, the server uses data analysis means to determine the visitor's requests and intentions.

[0810] The terminal can monitor the user's emotional state in real time through emotion recognition means. This involves advanced analytical techniques, such as analyzing voice tone and facial expressions. Based on the identified emotional state, the server adjusts its response to the visitor appropriately via response adjustment means. This allows for optimal support of the visitor without causing stress to the user.

[0811] As a practical application, if a suspicious person visits, the server can automatically reject the response using pre-configured responses. On the other hand, when a familiar delivery person visits, the terminal can recommend allowing direct interaction based on the emotion recognition results, demonstrating flexible responses tailored to the situation.

[0812] An example of a prompt would be: "We are currently developing an app that uses an emotion engine on your home's intercom to determine the visitor's intentions and suggest an appropriate response based on the resident's emotions. Please create a refined text description of this situation."

[0813] The hardware used includes smartphones and network-connected devices, while the software utilizes high-performance APIs for speech recognition. Furthermore, advanced computational processing, such as machine learning models, is required for sentiment analysis, and these processes enable the maintenance of sufficient response accuracy.

[0814] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0815] Step 1:

[0816] The server records the visitor's voice using voice acquisition equipment. The input is the visitor's voice, and the output is the digital data of that voice. In the voice acquisition process, a capture device such as a microphone is used, and the voice data is converted to an appropriate file format and saved.

[0817] Step 2:

[0818] The server converts digitized audio data into text format using a data conversion mechanism. The input is digital audio data, and the output is corresponding text data. Based on speech recognition technology, a process is performed to analyze each phoneme and convert it into characters.

[0819] Step 3:

[0820] The server analyzes text data to determine the visitor's requests and intentions using data analysis tools. The input is the converted text data, and the output is the analysis result regarding the visitor's intentions. Natural language processing techniques are used to extract important keywords and context to identify the content of the request.

[0821] Step 4:

[0822] The device detects the user's emotional state using emotion recognition technology. The input is the user's voice or video, and the output is the emotional data determined from it. An algorithm operates to analyze voice tone and facial expressions to infer the emotional state.

[0823] Step 5:

[0824] The server determines how to respond to the visitor using response adjustment mechanisms based on the acquired emotional data and analysis results. The input is the output from steps 3 and 4, and the output is the optimized response content for the visitor. This allows the system to automatically select the appropriate response action for the visitor according to their emotional state.

[0825] Step 6:

[0826] The user reviews the response displayed on the terminal and chooses whether or not to respond using the selection tool. The system presents response options as input, and the user's selection is the output. The options include direct response, refusal of response, and postponement of the response time.

[0827] Step 7:

[0828] The terminal manages interactions with visitors using dialogue support tools via a server, based on the user's selection. Input is the result of the user's selection, and output is the result of communication with the visitor. The response to the visitor is executed appropriately according to the selected response policy.

[0829] Step 8:

[0830] The server, if it determines that a visitor is suspicious, plays a pre-configured response via a response control mechanism. The input is the result of the suspicious person detection, and the output is the execution of the automated response. A safe response to the visitor is automatically played.

[0831] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0832] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0833] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0834] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0835] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0836] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0837] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0838] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0839] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0840] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0841] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0842] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0843] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0844] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0845] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0846] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0847] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0848] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0849] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0850] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0851] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0852] The following is further disclosed regarding the embodiments described above.

[0853] (Claim 1)

[0854] A means of acquiring the voice of a visitor,

[0855] A conversion means for converting the aforementioned audio into text data,

[0856] An analysis means that analyzes the aforementioned text data to determine the visitor's requirements,

[0857] Based on the aforementioned judgment, if the visitor is not suspicious, a means of notifying the resident of the requirements,

[0858] A means of selecting whether a resident will respond to a visitor according to the aforementioned requirements,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, further comprising a response means that enables direct dialogue between the resident and the visitor via an intercom when the resident permits response through the selection means.

[0862] (Claim 3)

[0863] The system according to claim 1, further comprising a response control means for playing a pre-set response to a visitor if, based on the aforementioned judgment, the visitor is determined to be suspicious.

[0864] "Example 1"

[0865] (Claim 1)

[0866] A means of acquiring the voice of a visitor,

[0867] A conversion means for converting the aforementioned audio into text data,

[0868] An analysis means that analyzes the aforementioned text data to determine the visitor's requirements,

[0869] Based on the aforementioned judgment, if the visitor is not suspicious, a means of notifying the resident of the requirements,

[0870] A means of selecting whether a resident will respond to a visitor according to the aforementioned requirements,

[0871] The notification means includes a method for displaying the requirements on the resident's mobile information terminal or dedicated terminal, and also enabling voice notification.

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, which includes means for enabling direct dialogue between the resident and the visitor via an interface when the resident permits the response through the selection means.

[0875] (Claim 3)

[0876] The system according to claim 1, which includes means for playing a pre-set response to a visitor and recording the visit history if the visitor is determined to be suspicious based on the aforementioned judgment.

[0877] "Application Example 1"

[0878] (Claim 1)

[0879] A means for acquiring the audio and images of visitors,

[0880] A conversion means for converting the aforementioned audio into text data,

[0881] An analysis means for analyzing the aforementioned text data and image data to determine the visitor's requirements and safety,

[0882] Based on the aforementioned determination, if the visitor is not suspicious, a control means is provided to notify the resident of the requirements and to control communication with the visitor.

[0883] A system including a means for residents to choose whether to respond to visitors according to the aforementioned requirements.

[0884] (Claim 2)

[0885] The system according to claim 1, which has a function to notify residents of the results of the safety assessment of visitors and to allow residents to decide whether or not to accept them.

[0886] (Claim 3)

[0887] The system according to claim 1, which has a response function that automatically initiates communication with visitors deemed safe and notifies residents of the reason for their visit.

[0888] "Example 2 of combining an emotion engine"

[0889] (Claim 1)

[0890] A voice acquisition method for acquiring the voice of a visitor,

[0891] The sound conversion means converts the aforementioned sound into text data,

[0892] A content evaluation means that analyzes the aforementioned text data to determine the visitor's requirements,

[0893] A means of acquiring emotions to recognize the emotions of residents,

[0894] A response adjustment means that adjusts response actions based on the emotional state of the resident obtained by the emotion acquisition means,

[0895] A selection presentation means that presents the adjusted response content to the resident and allows them to choose how to respond to the visitor,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] The system according to claim 1, further comprising a dialogue permission means that enables direct dialogue between the resident and the visitor via a communication device when the resident permits the response through the selection presentation means.

[0899] (Claim 3)

[0900] The system according to claim 1, further comprising a response control means for playing a pre-set response to a visitor if the requirements evaluation means determines that the visitor is suspicious.

[0901] "Application example 2 when combining with an emotional engine"

[0902] (Claim 1)

[0903] A voice acquisition method for acquiring the voice of a visitor,

[0904] A data conversion means for converting the aforementioned audio into text data,

[0905] A data analysis means that analyzes the aforementioned text data to determine the visitor's requirements,

[0906] A means of notifying residents of the requirements when it is determined that the visitor is safe based on the aforementioned judgment,

[0907] An emotion recognition method that recognizes the emotional state of residents in real time,

[0908] A response adjustment means that adjusts the response action to the visitor based on the aforementioned emotional state,

[0909] A means of selection by which a resident chooses whether to respond to a visitor according to the aforementioned requirements and emotional state,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, further comprising a dialogue support means that enables direct dialogue between the resident and the visitor via a communication device when the resident permits the response through the aforementioned choice means.

[0913] (Claim 3)

[0914] The system according to claim 1, further comprising a response control means that, if determined to be a suspicious visitor based on the aforementioned judgment, plays back a pre-set response to the visitor and reflects data from the emotion recognition means. [Explanation of symbols]

[0915] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring the voice of a visitor, A conversion means for converting the aforementioned audio into text data, An analysis means that analyzes the aforementioned text data to determine the visitor's requirements, Based on the aforementioned judgment, if the visitor is not suspicious, a means of notifying the resident of the requirements, A means of selecting whether a resident will respond to a visitor according to the aforementioned requirements, A system that includes this.

2. The system according to claim 1, further comprising a response means that enables direct dialogue between the resident and the visitor via an intercom when the resident permits response through the selection means.

3. The system according to claim 1, further comprising a response control means for playing a pre-set response to a visitor if, based on the aforementioned judgment, the visitor is determined to be suspicious.

Images