Judgment system, judgment method, and program

Abstract

Easily filter out messages that users might find offensive. [Solution] A determination system according to one embodiment of the present invention is a determination system that performs a determination regarding a message when creating a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user, and comprises a storage unit, a generation unit, and an output unit. The storage unit stores user information identified from the user's statement history. The generation unit inputs the user information, the message to be output to the user, and instruction information instructing the generation AI or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user using the user information, and causes the generation AI to generate the determination result. The output unit outputs the determination result.

Description

[[Technical Field]]

[0001] The present invention relates to a determination system, a determination method, and a program applicable to communication systems and the like that output messages to users. [[Background Art]]

[0002] In recent years, communication systems have been proposed in which interactive robots and the like are installed in residences to exchange messages with residents. Such systems are used, for example, in services that help alleviate loneliness among seniors and reduce life risks through conversations between robots and senior users.

[0003] For example, Patent Document 1 describes a system that outputs a message created by an operator to a user who is a monitoring target. In this system, an operator is caused to select a phrase group to be used for creating a message from a plurality of pre-stored phrase groups, and a message is created by changing a variable phrase region in a template using phrases included in the selected phrase group. This makes it possible to flexibly change the content of a message while reducing the labor of the operator. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2020-135799 [[Summary of Invention]] [[Problem to be Solved by the Invention]]

[0005] In a system like the one described above, it is expected that the user experience will improve by outputting messages that offer topics favorable to the user. On the other hand, depending on the content of the message, it is also possible that the user may find it offensive. Checking such messages requires considering various factors such as the user's characteristics and circumstances. Therefore, if, for example, an operator is creating the messages, the burden of checking them could become very large.

[0006] In view of the above circumstances, the object of the present invention is to provide a determination system, determination method, and program that can easily eliminate messages that may be offensive to the user. [Means for solving the problem]

[0007] To achieve the above objective, a determination system according to one embodiment of the present invention is a determination system that performs a determination regarding a message when creating a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user, and comprises a storage unit, a generation unit, and an output unit. The storage unit stores user information identified from the user's message history. The generation unit inputs the user information, the message to be output to the user, and instruction information that instructs the generation AI or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, using the user information, and causes the determination result to be generated. The output unit outputs the determination result.

[0008] This judgment system uses a generating AI or AI agent to produce a judgment result indicating whether a message scheduled to be output contains content that is not recommended for the user. The information used to generate the judgment result is user information identified from the user's message history. This makes it possible to judge messages scheduled to be output according to the user's characteristics, and to easily eliminate messages that may be offensive to the user.

[0009] The instruction information may include instructions to generate conclusion information indicating whether or not the message to be output is not recommended for the user as a result of the determination, and reason information explaining the reason for determining the conclusion indicated by the conclusion information based on the user information.

[0010] The user information may include at least one of the following: information relating to the user's preferences based on the user's statement history, or information relating to the user's physical condition based on the user's statement history.

[0011] The user information may include characteristic data that indicates the user's characteristics. In this case, the system may further include a management unit that deletes the characteristic data from the storage unit that indicates characteristics different from the user's current state, based on the user's statements or the time elapsed since the user information was registered.

[0012] The user information may include characteristic data that indicates the user's characteristics. In this case, if multiple sets of characteristic data indicating related characteristics are accumulated, the system may further include a management unit that registers the characteristic data indicating characteristics encompassing the related characteristics in the storage unit.

[0013] The determination system may further include an administrator terminal that accepts registration of the message by an administrator who manages the messages output to the user. In this case, the output unit may output the determination result to the administrator terminal.

[0014] A determination method according to one embodiment of the present invention is a determination method for making a determination regarding a message when creating a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user, wherein the computer system performs the following steps. A step of storing user information identified from the user's message history. A step of inputting the user information, the message to be output to the user, and instruction information that instructs the AI ​​or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, to generate the determination result. A step of outputting the aforementioned determination result.

[0015] A program according to one embodiment of the present invention causes a computer system that performs a determination regarding a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user to perform the following steps. A step of storing user information identified from the user's message history. A step of inputting the user information, the message to be output to the user, and instruction information that instructs the AI ​​or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, to generate the determination result. A step of outputting the aforementioned determination result. [Effects of the Invention]

[0016] As described above, the present invention makes it possible to easily eliminate messages that users may find offensive. The effects described herein are not necessarily limited, and any of the effects described in this disclosure may also be present. [Brief explanation of the drawing]

[0017] [Figure 1]It is a block diagram showing a configuration example of a communication system according to an embodiment of the present invention. [Figure 2] It is a diagram showing the hardware configuration of a center apparatus included in the communication system. [Figure 3] It is a schematic diagram showing an outline of the communication system. [Figure 4] It is a schematic diagram showing an example of user information. [Figure 5] It is a schematic diagram for explaining message determination processing. [Figure 6] It is a schematic diagram showing another example of a determination result regarding a message. [Figure 7] It is a schematic diagram showing an example of an update operation of a user information management unit. [Figure 8] It is a schematic diagram showing another example of an update operation of a user information management unit. [Figure 9] It is a schematic diagram showing an example of an integration operation of a user information management unit. [Figure 10] It is a flowchart showing an operation example of the communication system. [Figure 11] It is a schematic diagram showing an example of a message confirmation screen. [Figure 12] It is a schematic diagram showing an example of a message selection screen according to another embodiment. MODE FOR CARRYING OUT THE INVENTION

[0018] Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

[0019] [Outline of Communication System] FIG. 1 is a block diagram showing a configuration example of a communication system according to an embodiment of the present invention. The communication system 100 is a system that uses a user terminal 11 to greet and interact with a user 1. The present system is used, for example, for a monitoring system that targets elderly people and the like, and helps resolve loneliness and reduce life risks through communication with the user 1.

[0020] In the communication system 100 according to the present invention, a determination is made regarding the message when creating a message to be output from the user terminal 11 to user 1. In this embodiment, the communication system 100 is an example of a determination system.

[0021] As shown in Figure 1, the communication system 100 consists of a central device 10 installed in the management center, a plurality of user terminals 11 connected via a network to communicate with the central device 10, and an administrator terminal 12 connected to the central device 10 to communicate with it.

[0022] User terminal 11 is a terminal device installed in the home of a user (e.g., an elderly person) who is being monitored in this system. User terminal 11 is configured to acquire user 1's statements and output messages to user 1. Here, user 1's statements include the content that user 1 actually utters. In addition, content that user 1 inputs as their own statement using a predetermined input device is also included in user 1's statements.

[0023] For example, the user terminal 11 has acquisition means for acquiring statements such as utterances by user 1, and means for transmitting the acquired statements from user 1 to the central device 10 via the network. The user terminal 11 also has means for receiving messages transmitted from the central device 10 via the network and notified to user 1, and output means for notifying user 1 of those messages.

[0024] The acquisition means typically consists of a microphone M that collects the utterances of user 1. For example, user terminal 11 collects the voice spoken by user 1 to user terminal 11 with microphone M, converts it into a string using speech recognition technology, and transmits it to central device 10. An input device such as a touch panel may also be used as the acquisition means.

[0025] Considering that complex machine operation may be difficult for user 1, such as an elderly person, a speaker S is preferred as an output device that outputs sound. Alternatively, for user 1 whose hearing has deteriorated due to aging, a display device using a liquid crystal panel may be used so that the message can be recognized visually. In this way, the user terminal 11 has the function of conveying messages transmitted from the central device 10 to user 1 either audibly or visually.

[0026] The user terminal 11 only needs to have at least the above-mentioned components, but it may also be a conversational robot modeled after a small doll, so that it can be perceived by the user 1 as reminiscent of human interaction and feel familiar with.

[0027] Furthermore, when conveying a message to user 1 by voice, the user terminal 11 may be made to vibrate appropriately in accordance with the speech, or the eyes and mouth may be brightly displayed using a built-in light-emitting means (such as an LED).

[0028] The administrator terminal 12 is a device used by administrator 2 (operator), who resides at the management center and exchanges messages with user 1. For example, a PC capable of communicating with the center device 10 can be used as the administrator terminal 12. Alternatively, the center device 10 may be used directly as the administrator terminal 12.

[0029] In the communication system 100, the messages output from the user terminal 11 include a voice message and a response message.

[0030] A voice prompt message is a message output from the user terminal 11 at a scheduled date and time, and functions as a message to prompt user 1. The voice prompt message is generated in advance, for example, before the output date and time, and registered in a dedicated database. In this embodiment, the registration of the voice prompt message is performed by the message administrator 2 using the administrator terminal 12. If the message is output at the time it is sent to the user terminal 11, the message's sending date and time becomes the output date and time.

[0031] A response message is a message output from the user terminal 11 in response to a message from user 1 (for example, when user 1 speaks to the user terminal 11). For example, administrator 2 uses administrator terminal 12 to generate a message to respond to user 1 and registers it as a response message. The registered response message is sent to the user terminal 11 via the central device 10 and output to user 1.

[0032] Thus, the administrator terminal 12 is a terminal device that accepts message registrations from administrator 2, who manages the messages output to user 1. Note that message registration includes both pre-registering voice messages and generating and registering response messages as replies to user 1.

[0033] In this embodiment, the determination of the message to be output is mainly performed when creating a voice message, but it is not limited to this, and it is also possible to determine the content when creating a response message.

[0034] The central device 10 is a computing device that controls the operation of the entire communication system 100. For example, the central device 10 could be a server installed in a management center.

[0035] The center device 10 includes an input receiving unit 20, a user information management unit 21, a message generation processing unit 22, a message determination unit 23, and an output unit 24 as functional blocks. It also has control means for controlling these blocks. As will be described in detail later, these are implemented using appropriately known hardware and software stored in storage means.

[0036] A storage unit 25 is also connected to the center device 10. The storage unit 25 stores the user database (user DB 26). The user DB 26 is a database that stores user information for each user.

[0037] The input reception unit 20 receives the content of the operation input performed by administrator 2 on the administrator terminal 12. For example, the content of the operation input includes instructions, selections, and input data from administrator 2.

[0038] The user information management unit 21 manages user information stored in the user database 26 based on statements made by user 1 acquired at the user terminal 11. In this embodiment, the user information management unit 21 corresponds to the management unit.

[0039] The message generation processing unit 22 executes processes to generate a message to be output to user 1. These processes include processes for automatically generating a message and processes for generating a message according to the input from administrator 2.

[0040] The message determination unit 23 causes the generation AI 30 to generate a determination result indicating whether or not the message to be output to user 1 contains content that is not recommended for the user. This is a message determination process using the generation AI 30. In this embodiment, the message determination unit 23 corresponds to the generation unit.

[0041] The output unit 24 outputs the judgment result generated using the generation AI 30. In this embodiment, the output unit 24 displays the judgment result on the administrator terminal 12 so that the administrator 2 can check the judgment result.

[0042] The Generative AI 30 is an AI (Artificial Intelligence) that generates content such as text, images, audio, and video according to the input information. In this embodiment, a Generative AI capable of generating at least text is used, and for example, a Large Language Model (LLM) can be used.

[0043] As the generating AI 30, an external LLM built separately from the center device 10 may be used, or an LLM that has been independently trained and built may be used.

[0044] Alternatively, an AI agent may be used instead of the generating AI 30. The AI ​​agent is an AI that autonomously performs tasks, and for example, one that has the functionality of a generating AI can be used.

[0045] [Hardware configuration of the center equipment] Figure 2 shows the hardware configuration of the central device 10 of the communication system 100. As shown in the figure, the central device 10 includes a CPU (Central Processing Unit) 40, ROM (Read Only Memory) 41, RAM (Random Access Memory) 42, an input / output interface 43, and a bus 44 that connects these to each other.

[0046] The CPU 40 accesses RAM 42 and other memory as needed, performing various calculations and comprehensively controlling each block of the center unit 10. ROM 41 is a non-volatile memory in which the OS, programs, and firmware such as various parameters to be executed by the CPU 40 are permanently stored. RAM 42 is used as a working area for the CPU 40 and temporarily holds the OS, various running applications, and various data being processed.

[0047] Furthermore, the CPU 40 functions as one of the above-mentioned functional blocks (input reception unit 20, user information management unit 21, message generation processing unit 22, message determination unit 23, and output unit 24) configured in the center device 10.

[0048] The input / output interface 43 is connected to a display unit 45, an operation reception unit 46, a communication unit 47, and a storage unit 25, etc. The display unit 45 is a display device using, for example, an LCD (Liquid Crystal Display), an OLED (Organic Electro Luminescence Display), a CRT (Cathode Ray Tube), etc. The operation reception unit 46 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. The communication unit 47 is, for example, a NIC (Network Interface Card) for Ethernet® or various modules for wireless communication such as wireless LAN, and is responsible for communication processing between the user terminal 11 and the administrator terminal.

[0049] The storage unit 25 is a non-volatile memory such as an HDD (Hard Disk Drive), flash memory (SSD; Solid State Drive), or other solid-state memory. In addition to the OS mentioned above, the storage unit 25 stores schedule information, various flags, parameters, and software that implements each functional block of the center device 10.

[0050] In this embodiment, the storage unit 25 stores data such as the user DB 26 shown in Figure 1, applications for interaction with the user terminal 11 including message generation, other programs, and databases.

[0051] The hardware configuration of the user terminal 11 and the administrator terminal 12 is basically the same as the hardware configuration of the central device 10 described above. Each user terminal 11 has a microphone M as an acquisition means and a speaker S as an output means on its front, as described above.

[0052] [Overview of the communication system] Figure 3 is a schematic diagram showing an overview of the communication system 100. In the communication system 100, the message generation process by the message generation processing unit 22, the message determination process by the message determination unit 23, and the message confirmation process performed by administrator 2 using administrator terminal 12 are executed in this order. Figure 3 shows the flow of these processes.

[0053] The message generation process is the process of generating a message (scheduled output message 35) that is to be output from user terminal 11 to user 1. The message generation processing unit 22 executes arbitrary processes to generate a message for user 1, and finally outputs the scheduled output message 35.

[0054] The scheduled output message 35 can be generated in various ways. For example, the scheduled output message 35 may be generated using a generation AI or the like, with a topic selected by administrator 2 from among several topics as input. Alternatively, the scheduled output message 35 may be generated by administrator 2's selection operation (for example, selecting a phrase from a group of phrases), as described in Patent Document 1. Furthermore, a message directly entered by administrator 2 may be read as the scheduled output message 35. In addition, the scheduled output message 35 may be generated fully automatically using a generation AI or the like, without the intervention of administrator 2.

[0055] The message judgment process determines whether the content of the planned message 35 is undesirable for user 1 before outputting it to user 1. Here, undesirable content for user 1 refers to content that user 1 may find offensive. Such messages should not be output to user 1; they are what are known as "NG" messages.

[0056] The message determination unit 23 inputs user information 34 identified from user 1's speech history, a message to be output to user 1 (scheduled output message 35), and message determination instruction information that instructs the generation AI to generate a determination result 36 indicating whether or not the scheduled output message 35 is not recommended for user 1 using the user information 34, and causes the AI ​​to generate the determination result 36.

[0057] User information 34 is information generated based on user 1's past statement history, and is generated for each user and stored in user DB 26. User information 34 includes, for example, information (statements) extracted from the user's past statements. For example, statements from user 1 that indicate user 1's preferences or current situation are stored in association with user 1's ID. In Figure 3, user information 34 for user 1, whose user ID is "001", is read from user DB.

[0058] Message determination instruction information is information describing instructions for the generating AI 30, and typically uses text data written in natural language called a prompt. The message determination instruction information includes an instruction to output a determination result 36 in a predetermined format indicating whether the output message 35 is not recommended for user 1. The determination result 36 is typically generated to explicitly indicate whether the output message 35 is recommended or not, but the format of the determination result 36 can be arbitrarily specified (see Figures 5 and 6, etc.). In this embodiment, the message determination instruction information corresponds to instruction information.

[0059] In this way, the message judgment process inputs user information 34 and the planned output message 35 to the generation AI 30 (LLM), and determines whether the planned output message 35 contains undesirable content, i.e., whether the planned output message 35 is an NG message. This process is performed for each user and each message, and a judgment result 36 is generated for each. Once the judgment result 36 is generated, the output unit 24 outputs the judgment result 36 to the administrator terminal 12.

[0060] The message confirmation process presents the judgment result 36 to administrator 2 for review and allows administrator 2 to decide whether or not to output the scheduled message 35 to user 1. This allows administrator 2 to, for example, identify scheduled messages 35 that are not recommended for user 1 and reject such messages.

[0061] On the administrator terminal 12, for example, the scheduled output message 35 is displayed along with its judgment result 36 (see Figure 11, etc.). Administrator 2 refers to these judgment results 36 to decide whether or not to make the scheduled output message 35 an actual message, and inputs the decision result via the administrator terminal 12. For example, if the judgment result 36 indicates that it is not recommended, administrator 2 can carefully consider whether or not to output the scheduled output message 35.

[0062] For example, if administrator 2, after referring to the judgment result 36, decides to output the scheduled output message 35 to user 1, the scheduled output message 35 is registered as the message (output message 37) that will actually be output to user 1 at the predetermined output date and time. Thus, in this embodiment, the final decision on the output message 37 is made by administrator 2.

[0063] On the other hand, if administrator 2, after referring to the judgment result 36, decides not to output the scheduled message 35 to user 1, the scheduled message 35 is rejected. In this case, the scheduled message 35 may be generated again by the message generation process. Also, for example, if multiple scheduled messages 35 are generated, the message that was not judged as unrecommended in the judgment result 36 (i.e., the message judged to be outputtable) may be selected as the output message 37 (see Figure 12).

[0064] [User Information] Next, we will explain the specific contents of the user information 34 stored in the user database 26. Figure 4 is a schematic diagram showing an example of user information. User information 34 is stored in the user database 26 as text data, for example, along with identifiers such as the user ID.

[0065] As shown in Figure 4, the user information 34 includes multiple characteristic data 50 that represent the characteristics of user 1. Each characteristic data 50 is data that represents characteristics such as user 1's preferences or current situation. The characteristic data 50 includes, for example, information extracted from user 1's statements, information summarized from user 1's statements, and information generated from user 1's statements.

[0066] Next, we will specifically explain the contents of user information 34 (characteristic data 50). User information 34 includes information about user 1's preferences based on user 1's statement history. In this embodiment, we will explain the case where user information 34 mainly uses information about user 1's negative preferences. Here, information about user 1's negative preferences refers to things that user 1 finds difficult or dislikes, or things that user 1 has a negative impression of. This can also be rephrased as things that user 1 finds difficult, repulsive, or avoidable. In the example shown in Figure 4, statements such as "I want to avoid crowds" and "I don't like sticky foods" correspond to information about user 1's negative preferences.

[0067] Information regarding negative preferences serves as a basis for determining whether user 1 will have a negative impression of the content of a message. This makes it possible to appropriately identify messages that may be unpleasant for user 1. As a result, it is possible to avoid situations such as displaying messages that include topics that user 1 dislikes, such as food.

[0068] User information 34 also includes information about User 1's physical condition based on User 1's speech history. Here, information about User 1's physical condition refers to information that describes User 1's physical condition when outputting a message, and is typically medical information (including chronic illnesses and past medical history) related to illness, injury, or other health conditions. In the example shown in Figure 4, a statement such as "I sprained my ankle" corresponds to information about User 1's physical condition.

[0069] Information regarding the user's physical condition functions as information indicating user 1's situation, such as whether or not user 1's activities are restricted, and the types of activities that are restricted for user 1. This makes it possible to determine that messages containing things that are restricted for user 1 are not recommended. As a result, situations such as displaying messages about running to user 1 who has sprained their ankle can be avoided.

[0070] In this embodiment, the user information management unit 21, as described with reference to Figure 1, manages multiple characteristic data 50 included in user information 34 based on the user 1's speech history. The management of characteristic data 50 by the user information management unit 21 includes collecting, deleting, and generating characteristic data 50.

[0071] Specifically, the user information management unit 21 detects information regarding user 1's negative preferences and information regarding user 1's physical condition from the statements made by user 1 acquired by user terminal 11, and registers this information as characteristic data 50 in user information 34. In other words, the user information management unit 21 acquires information about things user 1 dislikes and user 1's medical information from user 1's statements during conversations between user 1 and user terminal 11, and sequentially stores this information as user information 34.

[0072] Information regarding negative preferences can be detected, for example, from negative words ("I want to avoid," "I'm not good at it," "I hate it," etc.). Similarly, information regarding physical condition can be detected from medical-related words ("sprain," "cold," "high blood pressure," etc.). Alternatively, machine learning or other techniques may be used to identify things that User 1 has a negative impression of or User 1's physical condition from the entire conversation.

[0073] This information is registered as characteristic data 50 along with the date of the statement. This creates user information 34 based on user 1's statement history. Furthermore, user information 34 is updated sequentially as new statements are made by user 1 or as time passes.

[0074] The method for obtaining the characteristic data 50 and the types of characteristic data 50 are not limited. For example, information regarding negative preferences or information regarding user 1's physical condition may be detected and registered. Other types of information may also be used as characteristic data 50. For example, information regarding user 1's experiences (accidents, failures, etc.) may be registered. In addition, characteristic data 50 declared by user 1 may be registered in advance as user information 34. For example, information regarding negative preferences or physical condition, or information representing the user's attributes (age, gender, place of residence, etc.) may be registered as characteristic data 50.

[0075] [Message detection process] Figure 5 is a schematic diagram illustrating the message judgment process. Figure 5(a) is an example of input information 51 to be input to the generating AI 30. Figure 5(b) is an example of a judgment result 36 output from the generating AI 30.

[0076] The message determination instruction information 52 indicates an instruction to generate a determination result 36 indicating whether or not the output message 35 contains content that is not recommended for user 1. The content of the message determination instruction information 52 is recorded, for example, as a predetermined template and read as appropriate when generating the input information 51.

[0077] Here, at the beginning of the message judgment instruction information 52 (the text above "1. Classification Example"), two pieces of information are listed separated by a blank column. The upper piece of information describes the prerequisites 53, which explain the setting and situation of a fictional character assumed to be the speaker of the planned output message 35. The lower piece of information describes the specific judgment instruction 54 for generating the judgment result 36.

[0078] In judgment instruction 54, the system is instructed to generate a judgment result 36 that classifies whether the planned output message 35 ("the message to be sent") can be sent as either "yes" or "no," based on the user information 34 ("#Previous statements").

[0079] Furthermore, the message determination instruction information 52 includes a classification example 55 ("1. Classification Example"). Classification example 55 is a description based on a technique called Few-Shot prompting, and it is a description that provides several output examples in order to obtain a more desired output result.

[0080] Classification Example 55 includes examples for User Information 34 ("#Previous Statements"), Planned Output Message 35 ("#Message to be Sent"), and Judgment Result 36 ("Reason for Judgment" and "Send Approval / Disapproval"). By including such examples, it becomes possible to generate Judgment Result 36 in the same format as Classification Example 55.

[0081] Here, as classification example 55 for judgment result 36, "reason for judgment" and "transmission possible / not possible" are described. "Transmission possible / not possible" is an example of conclusion information that shows the conclusion of the judgment, and "reason for judgment" is an example of reason information that shows the reason for reaching the conclusion (see Figure 5(b)). Describing conclusion information and reason information in classification example 55 is equivalent to describing the instruction to output conclusion information and reason information as judgment result 36.

[0082] Thus, the message judgment instruction information 52 includes an instruction to generate a judgment result 36 which includes conclusion information indicating whether or not the scheduled output message 35 contains content that is not recommended for user 1, and reason information that explains the reason for determining the conclusion indicated by the conclusion information based on user information. This makes it possible to present not only the conclusion, such as "send or not," but also the reason for it to administrator 2.

[0083] In the input information 51, the text following the message judgment instruction information 52 (the text below "2. Problem") contains user information 34 and the planned output message 35. User information 34 is listed under the item "Previous statements," with each characteristic data 50 listed in bullet points. The planned output message 35 is listed under the item "#Message to be spoken next." Here, the planned output message 35 is "Okra is good for your health." The input information 51 generated in this way is input to the generating AI 30 by the message judgment unit 23.

[0084] Figure 5(b) shows the judgment result 36 generated by the generating AI 30, which received the input information 51 shown in Figure 5(a). This judgment result 36 is output to the administrator terminal 12 by the output unit 24.

[0085] The judgment result 36 includes the reason information 56, "reason for judgment," and the conclusion information 57, "transmission feasibility." If the transmission feasibility is "no," it indicates that the planned output message 35 should not be sent, meaning it is not recommended for user 1. Conversely, if the transmission feasibility is "yes," it indicates that the planned output message 35 can be sent, meaning it is not not recommended for user 1.

[0086] Reason information 56 describes the reason for determining the conclusion shown in conclusion information 57. This reason is basically based on user information 34. In this case, user information 34 contains the statement, "I don't like sticky foods." Based on this, the "sendability" of the message "Okra is good for your health" is determined to be "No."

[0087] In this way, by using user information 34 as a basis, it is possible to accurately determine whether or not the planned output message 35 is not recommended. Furthermore, by outputting both conclusion information 57 and reason information 56, it is possible to adequately support the administrator 2's decision on whether or not to register it as the output message 37.

[0088] Figure 6 is a schematic diagram showing another example of a judgment result regarding a message. Figure 6 shows the judgment result 36 shown in Figure 5(b) with the supporting information 58 added.

[0089] The supporting information 58 is an example of reasoning information 56, and is the information that forms the basis for determining the conclusion shown in the conclusion information 57, among the multiple pieces of information (individual characteristic data 50) included in the user information 34. In this case, the supporting information is the statement made by user 1 ("I don't like sticky foods") registered on 2025 / 1 / 1 in the user information 34 shown in Figure 5(a). In this case, the determination result 36 will include the content of user 1's statement and the date as supporting information 58.

[0090] Thus, the message determination instruction information 52 includes an instruction to present the basis information 58 that serves as the basis for the reason information 56 from the user information 34. This can be achieved, for example, by including an example of reason information 56 in the classification example 55. Alternatively, it can be achieved by including an instruction in the initial instruction to the effect of, for example, "include the data that serves as the basis for the determination."

[0091] For example, if the characteristic data 50 used as the basis is old, its content may not be suitable for the current judgment. Administrator 2 can take these points into consideration to interpret the judgment result 36 more appropriately and make a more appropriate decision on whether or not to change the planned output message 35.

[0092] [Operation of the User Information Management Unit] The following describes how the user information management unit 21 manages user information 34.

[0093] Figure 7 is a schematic diagram showing an example of the update operation of the user information management unit 21. Figures 7(a) and 7(b) show the user information 34 before and after the update. In this case, the user information management unit 21 deletes characteristic data 50 that indicates characteristics different from the current state of user 1, based on user 1's statement 5.

[0094] For example, as shown in Figure 7(a), if user 1 makes the statement "I don't like spicy food" 5, the characteristic data "I don't like spicy food" 50 is registered in user information 34, as this statement is considered to indicate a negative preference.

[0095] After registering such characteristic data 50, suppose user 1 makes the statement 5, "I can now eat spicy food," as shown in Figure 7(b). In this case, the characteristic "I don't like spicy food" is different from user 1's current characteristics. Therefore, the previously registered characteristic data 50 "I don't like spicy food" is deleted from user information 34. In this way, user information 34 is updated based on the new characteristics in response to changes in user 1's preferences. This makes it possible to appropriately determine the output message 35 according to user 1's current characteristics.

[0096] Figure 8 is a schematic diagram showing another example of the update operation of the user information management unit 21. Figures 8(a) and 8(b) show the user information 34 before and after the update. Here, the user information management unit 21 deletes characteristic data 50 from the storage unit 25 that indicates characteristics different from the current status of user 1, based on user 1's statements or the time elapsed since user information 34 was registered. This process is mainly applied to characteristic data 50 that indicates user 1's physical condition, such as medical information.

[0097] Figure 8(a) shows that characteristic data 50, "I sprained my ankle," is registered as a statement made by User 1 on 2025 / 1 / 10. Note that the user information 34 shown in Figure 8(a) is as of 2025 / 1 / 20. At this point, only about 10 days have passed since the characteristic data 50 indicating that User 1 sprained their ankle was registered, and it is possible that the sprain has not yet fully healed. Therefore, the characteristic data 50, "I sprained my ankle," is maintained.

[0098] Figure 8(b) shows user information 34 as of 2025 / 2 / 10. At this point, one month has passed since the date the characteristic data 50 indicating that user 1 sprained their ankle was registered. Therefore, it is highly likely that the sprain has completely healed. Consequently, the characteristic data 50, "I sprained my ankle," is likely to be different from user 1's current condition.

[0099] Therefore, the user information management unit 21 deletes the characteristic data 50, "I sprained my ankle," as shown in Figure 8(b). This allows the provision of messages, such as those related to running, to resume. The period for deleting characteristic data 50 is not particularly limited, but may be set appropriately depending on the type of injury, for example. For example, it may be possible to set it to delete after 10 days in the case of a "burn" and after 2 months in the case of a "fracture."

[0100] Figure 9 is a schematic diagram showing an example of the integrated operation of the user information management unit. Figures 9(a) and 9(b) show the user information 34 before and after the update. Here, when the user information management unit 21 has accumulated multiple characteristic data 50 that indicate related characteristics, the characteristic data 50 that indicates the characteristics encompassing the related characteristics is registered in the storage unit 25.

[0101] Here, the characteristic data 50 that shows related characteristics refers to characteristic data that includes things belonging to the same genre (category), for example. Here, the characteristic data 50 for "baseball," "soccer," and "basketball" are shown in order from top to bottom. These characteristic data 50 can be said to belong to the genre of "ball games."

[0102] Thus, when a certain number (for example, three or more) of characteristic data 50 indicating related characteristics are accumulated, it is highly likely that user 1 has a negative impression of matters in that genre. For this reason, the user information management unit 21 generates characteristic data 50 "dislikes ball games" as information that integrates the accumulated characteristic data 50 of the same genre.

[0103] Here, if the statement "I hate basketball" (trait data 50) is recorded, a certain number of trait data 50 indicating related traits have been accumulated, and trait data 50 "I hate ball games" is generated.

[0104] This makes it possible to avoid potentially offending User 1 by bringing up a topic such as "volleyball." In other words, it can suppress the output of messages that are assumed to be disliked or uncomfortable for User 1.

[0105] In Figure 9, if User 1's statement history includes a statement saying "I like tennis," it is clear that they do not dislike ball games in general, so the trait data 50 "Dislikes ball games" is not generated. Also, if a statement saying "I like tennis" is made after the trait data 50 "Dislikes ball games" has been generated, the trait data 50 "Dislikes ball games" may be deleted, and the genre merge may be reversed.

[0106] [Operation flow of the communication system] Figure 10 is a flowchart illustrating an example of the communication system's operation. Figure 10 shows a series of operations leading up to the creation of a voice message. This section describes the operations, including the message determination process performed in the central device 10.

[0107] First, the message generation processing unit 22 creates the scheduled output message 35 (step 101). The scheduled output message 35 is created as a message to user 1, for example, after setting the target user 1 and the output date and time. The method for creating the scheduled output message 35 is not limited. For example, any method used in the message generation process described with reference to Figure 3 can be adopted.

[0108] Next, a determination process is performed for the scheduled output message 35 (step 102). Specifically, the message determination unit 23 reads the scheduled output message 35 and the user information 34 of user 1 to whom it will be output. Input information 51 (see Figure 7) is generated from this information and input information 51 is input to the generation AI 30. As a result, the generation AI 30 generates a determination result 36 indicating whether or not the scheduled output message 35 contains content that is not recommended for user 1.

[0109] Next, the output unit 24 outputs the judgment result 36 for each target topic Tt to the administrator terminal 12 (step 103). The administrator terminal 12 displays the judgment result 36 using a predetermined GUI (see Figure 11). Administrator 2 refers to the judgment result 36 and decides whether or not to adopt (register) the scheduled output message 35, and inputs that decision into the administrator terminal 12.

[0110] Next, it is determined whether or not to adopt the planned output message 35 (step 104). Specifically, the input reception unit 20 reads the input information regarding whether or not the planned output message 35 entered by administrator 2 on the administrator terminal 12 is adopted, and a decision is made on whether or not to adopt it according to the input information.

[0111] For example, if administrator 2 inputs that they do not want to use the planned output message 35 (No in step 104), the planned output message 35 is rejected. Then, the process returns to step 101, a new planned output message 35 is generated, and the same judgment process is performed.

[0112] Furthermore, if administrator 2 inputs that they will adopt the planned output message 35 (Yes in step 104), the planned output message 35 is registered as the message actually output to user 1 (output message 37 in Figure 3) (step 105).

[0113] Figure 11 is a schematic diagram showing an example of a message confirmation screen. The message confirmation screen 60 is a screen for administrator 2 to check the scheduled message 35 and decide whether or not to send it. The message confirmation screen 60 displays the scheduled message 35, the judgment result 36 (conclusion information 57, reason information 56, and justification information 58), the setting destination information 61, a message change button 62, and a message registration button 63.

[0114] The destination information 61 is information about the destination of the scheduled output message 35 (the message sending slot provided for each user and each sending date and time). The destination information 61 includes information about user 1 (group name, username, user ID, etc.), sending date and time, existing settings, etc. The existing settings indicate whether another message is set in the sending slot described in the destination information 61.

[0115] Administrator 2 can, for example, check the conclusion information 57 to see the conclusion regarding whether the transmission is permitted or not obtained through the judgment process. Here, the "×" sign indicates that the conclusion of the judgment process is "No". If the conclusion of the judgment process is "Permitted", a sign such as "○" or "OK" will be displayed.

[0116] Administrator 2 can also check the reason why the judgment is "no" by looking at the reason information 56. Here, the user information 34 that forms the basis of the reason information 56 is also displayed. This makes it possible to properly interpret the judgment result 36.

[0117] The message change button 62 is used to change the scheduled output message 35. For example, if administrator 2, after reviewing the judgment result 36, decides not to adopt the scheduled output message 35, they can reject the current scheduled output message 35 by pressing the message change button 62 (No. in step 104 of Figure 10). In this case, for example, a new scheduled output message 35 is generated, and its judgment result 36 is displayed on the message confirmation screen 60. If there are multiple candidates for the scheduled output message 35, the judgment results 36 for the other candidates may also be displayed.

[0118] Administrator 2 can also register the scheduled message 35 as a message to be sent by pressing the message registration button 63. If another message is already set in the target sending slot (if there is an existing setting), the latest setting will overwrite it. This allows Administrator 2 to send a message to User 1 using a message that they have confirmed is ready to send.

[0119] In the communication system 100 according to this embodiment, the generation AI 30 is used to generate a determination result 36 indicating whether or not the scheduled output message 35 contains content that is not recommended for user 1. User information 34, identified from user 1's speech history, is used as the information for generating the determination result 36. This makes it possible to determine the scheduled output message 35 according to user 1's characteristics, and to easily eliminate messages that user 1 may find offensive.

[0120] In systems that use communication robots or similar devices to communicate with users, the user experience can be improved by outputting messages with appropriate topics. However, since each user has different tastes and preferences, there is a possibility of sending messages that users may find offensive. Furthermore, if messages are created manually by an operator, it is anticipated that the operator will be burdened with tasks such as understanding what words are considered inappropriate by users.

[0121] In contrast, in the communication system 100 according to this embodiment, user information 34 based on user 1's speech history is input to the generating AI 30 to determine whether or not the planned output message 35 contains undesirable content. This makes it easy for administrator 2 to determine messages that should not be sent to user 1. As a result, it becomes possible to sufficiently prevent causing offense to user 1.

[0122] <Other Embodiments> The present invention is not limited to the embodiments described above, and various other embodiments can be realized.

[0123] In the above embodiment, a configuration was described in which administrator 2 checks one scheduled output message 35 at a time. However, the present invention is not limited to this, and may be applied to configurations in which administrator 2 can check multiple scheduled output messages 35 simultaneously.

[0124] For example, it is possible to automatically generate scheduled output messages 35 for multiple output dates and times (settings). For instance, administrator 2 can set multiple dates (or periods) and multiple time slots for each date, thereby setting multiple output dates and times. Multiple scheduled output messages 35 corresponding to these output dates and times are generated as needed using a generation AI or similar.

[0125] In this case, a screen for checking multiple scheduled output messages 35 is displayed on the administrator terminal 12. The message determination unit 23 also performs determination processing on the multiple scheduled output messages 35, and the determination result 36 is displayed along with the scheduled output messages 35 for confirmation.

[0126] Figure 12 is a schematic diagram showing an example of a message selection screen according to another embodiment. The message selection screen 65 shown in Figure 12 is a screen that displays multiple scheduled output messages 35 that are automatically generated corresponding to each scheduled output date and time when multiple output dates and times are set. Here, it is assumed that the message determination unit 23 has already performed determination processing on the multiple scheduled output messages 35. Figure 12(a) shows the message selection screen 65 in its initial state. Figure 12(b) shows the message selection screen 65 displayed while the administrator 2 is working.

[0127] The message selection screen 65 has multiple message display bars 66, one for each of the multiple scheduled output messages 35. Each message display bar 66 has setting destination information 67, topic T, scheduled output message 35, a setting button 68, and a cancel button 69. The setting destination information 67 shows a different date and time for each message display bar 66. The target user 1 is the same for all of them. The setting button 68 is for setting (approving) the scheduled output message 35 as a prompt message. The cancel button 69 is for canceling (rejecting) the scheduled output message 35.

[0128] As shown in Figure 12(b), for example, when setting button 68 (in this case, the setting button 68 for "History of Cherry Blossoms") is selected, setting button 68 is highlighted, and the message display bar 66 is displayed in a lighter color overall. This indicates that the scheduled output message 35 related to "History of Cherry Blossoms" has been set as the prompt message.

[0129] Furthermore, as shown in Figure 12(b), if you select, for example, the cancel button 69 (in this case, the cancel button 69 for "bamboo shoots"), the cancel button 69 will be highlighted, and the message display bar 66 will be displayed in a lighter color overall. This indicates that the scheduled output message 35 related to "bamboo shoots" has been canceled.

[0130] Additionally, a warning mark 70 is displayed on the message selection screen 65. The warning mark 70 indicates that the scheduled output message 35 has been determined to contain content that is not recommended for user 1. This allows administrator 2 to easily identify the scheduled output message 35 that has been determined to be problematic through the evaluation process.

[0131] Furthermore, when Administrator 2 selects the warning mark 70, the details of the judgment result 36 (for example, the message confirmation screen 60 shown in Figure 11) are displayed. This allows Administrator 2 to check the reasons why the planned output message 35 is not recommended, and to make an appropriate decision on whether or not to use the planned output message 35. Note that the message change button 62 and message registration button 63 shown in Figure 11 may be linked to the cancel button 69 and setting button 68 shown in Figure 12. For example, when the message change button 62 is pressed in Figure 11, the cancel button 69 is selected in Figure 12.

[0132] As shown in Figure 12(b), when the scheduled output message 35 is canceled, a new scheduled output message 35 corresponding to the same output date and time is generated as appropriate and added to the message selection screen 65. A judgment process is also performed on the new scheduled output message 35, and if it is determined that its content is not recommended, the warning mark 70 is displayed again. This makes it easy for administrator 2 to check the new scheduled output message 35 for the output date and time when the message was canceled.

[0133] In the above embodiment, we mainly described an example where message determination is made based on information about user 1's negative preferences, but message determination may also be made based on information about positive preferences. For example, if user 1's favorite sports team is registered as a positive preference, displaying a message about a rival team may make user 1 feel uncomfortable. In this way, it is also possible to determine whether or not content is not recommended based on positive preferences.

[0134] In the above embodiment, the message to be output is determined for each user 1. However, users 1 who share the same user information 34 may be registered as members of a group related to that user information 34, and the message to be determined on a group basis may be performed.

[0135] The above embodiment describes a case where an administrator performs a final confirmation of a message before it is registered to be output to a user. The present invention can also be applied to systems that do not involve selection or decision by an administrator when creating messages. For example, it can determine whether or not several automatically generated messages scheduled for output contain undesirable content. If a message is determined to be undesirable, a new message is generated. Messages that are not determined to be undesirable (messages determined to be outputtable) are automatically registered as messages to be actually output. Even with such a configuration, by using the message evaluation results, it becomes possible to easily eliminate messages that users may find offensive.

[0136] It is also possible to combine at least two of the feature features of the present technology described above. In other words, the various feature features described in each embodiment may be combined arbitrarily, regardless of the specific embodiment. Furthermore, the various effects described above are merely examples and not limiting, and other effects may also be exhibited.

[0137] A judgment system according to one embodiment of the present invention can contribute to solving social issues such as loneliness and isolation among the elderly, extending healthy life expectancy, improving the quality of life (QoL) of the elderly, and the decline in the working-age population. Furthermore, a judgment system according to one embodiment of the present invention can contribute to achieving Goal 3 of the Sustainable Development Goals (SDGs) adopted by the United Nations, "Ensure healthy lives and promote well-being for all." [Explanation of symbols]

[0138] 1…User 2…Administrator 10... Center device 11…User terminal 12…Administrator terminal 21…User Information Management Department 23...Message determination unit 24…Output section 25...Storage section 30…Generation AI 34…User Information 35… Scheduled output message 36…Judgment result 52...Message judgment instruction information 100... Communication System

Claims

1. A determination system that performs a determination regarding a message when creating a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user, A storage unit that stores user information identified from the user's message history, A generation unit inputs the user information, the message to be output to the user, and instruction information that instructs the generation AI or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, and causes the generation unit to generate the determination result. Output unit that outputs the judgment result A judgment system equipped with the following features.

2. A determination system according to claim 1, The instruction information includes instructions to generate conclusion information indicating whether or not the message to be output is undesirable for the user as a result of the determination, and reason information explaining the reason for determining the conclusion indicated by the conclusion information based on the user information. Judgment system.

3. A determination system according to claim 1, The user information includes at least one of the following: information relating to the user's preferences based on the user's statement history, or information relating to the user's physical condition based on the user's statement history. Judgment system.

4. A determination system according to claim 1, The user information includes characteristic data that indicates the characteristics of the user, The system further comprises a management unit that deletes characteristic data from the storage unit that shows characteristics different from the current status of the user, based on the user's statements or the time elapsed since the user information was registered. Judgment system.

5. A determination system according to claim 1, The user information includes characteristic data that indicates the characteristics of the user, When multiple sets of characteristic data representing related characteristics are accumulated, the system further comprises a management unit that registers the characteristic data representing characteristics encompassing the related characteristics in the storage unit. Judgment system.

6. The determination system according to claim 1, further, The system includes an administrator terminal that accepts registration of the message by an administrator who manages the messages output to the user, The output unit outputs the determination result to the administrator terminal. Judgment system.

7. A determination method for creating a message to be output to a user from a user terminal capable of acquiring user statements and outputting messages to the user, wherein the determination is made regarding the message, A step of storing user information identified from the user's message history, The steps include inputting the user information, the message to be output to the user, and instruction information instructing the AI ​​or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, thereby causing the AI ​​to generate the determination result; The step of outputting the aforementioned determination result A method for determining whether to execute a decision on a computer system.

8. When creating a message to be output to a user from a user terminal that is capable of acquiring user statements and outputting messages to the user, the computer system that makes decisions regarding the message, A step of storing user information identified from the user's message history, The steps include inputting the user information, the message to be output to the user, and instruction information instructing the AI ​​or AI agent to generate a determination result indicating whether or not the message to be output is not recommended for the user, thereby causing the AI ​​to generate the determination result; The step of outputting the aforementioned determination result A program that executes the command.

Images