User support device, user support method, and program

Abstract

To provide a user support device, user support method, and program that can improve the satisfaction of users of the service. [Solution] The image acquisition unit of the user assistance device of this disclosure acquires image information including the user using the service. The identification unit identifies the user based on the image information. The location acquisition unit acquires the location information of the identified user. The object output unit outputs a control signal to cause the user's dedicated concierge object information to be output in the display area corresponding to the user's location information among a plurality of display areas along the user's movement path. The instruction acquisition unit acquires the user's instruction information, including natural language. The response output unit uses a large-scale language model to generate a response from the dedicated concierge based on the user's instruction information and outputs a control signal to the output device to execute processing corresponding to the dedicated concierge's response.

Description

Technical Field

[0001] The present disclosure relates to a user support apparatus, a user support method, and a program.

Background Art

[0002] In facilities such as smart retail stores and restaurants, there is a demand for improving the satisfaction of users who use services at the facilities.

[0003] Patent Document 1 discloses a technology that displays a user-dedicated concierge on a kiosk terminal, and provides services such as procedure guidance by the concierge to the user using the kiosk terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The inventors consider that when a user can realize an experience in which a dedicated concierge accompanies the user as part of a service, improvement in the satisfaction of the user using the service can be expected.

[0006] However, in Patent Document 1, the dedicated concierge cannot follow the user when the user moves to another location. Therefore, Patent Document 1 has a problem that there is room for improvement in enhancing the satisfaction of users who use the service.

[0007] In view of such problems, the present disclosure aims to provide a user support apparatus, a user support method, and a program that can improve the satisfaction of users who use services.

Means for Solving the Problems

[0008] The user support device of this disclosure is An image acquisition unit that acquires image information including the user of the service, An identification unit that identifies the user based on the image information, A location acquisition unit that acquires the location information of the identified user, An object output unit outputs a control signal to output object information of the user's personal concierge to a display area linked to the location information corresponding to the user's location information, among a plurality of display areas along the user's movement path. An instruction acquisition unit that acquires the user's instruction information, The system includes a response output unit that uses a large-scale language model to generate a response from the dedicated concierge based on the user's instruction information, and outputs a control signal to an output device to execute processing corresponding to the dedicated concierge's response.

[0009] The user support methods described in this disclosure are: Computers Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, The location information of the identified user is obtained, A control signal is output to the display area linked to the location information corresponding to the user's location information among the multiple display areas along the user's movement path, causing the object information of the user's dedicated concierge to be displayed. The user's instruction information is obtained, Using a large-scale language model, the system generates a response from the dedicated concierge based on the user's instructions and outputs a control signal to an output device to execute the processing corresponding to the dedicated concierge's response.

[0010] The program disclosed herein is Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, Acquire the identified location information of the user, output a control signal for outputting the object information of the user's dedicated concierge to a display area associated with location information corresponding to the user's location information, among a plurality of display areas on the user's movement route, Acquire the user's instruction information, Using a large language model, generate a response from the dedicated concierge based on the user's instruction information, and cause a computer to execute a process of outputting a control signal for executing processing corresponding to the dedicated concierge's response to an output device.

Effects of the Invention

[0011] According to the present disclosure, it is possible to provide a user support device, a user support method, and a program that can improve the satisfaction of users who use services.

Brief Description of the Drawings

[0012] [Figure 1] 1 is a block diagram illustrating an example configuration of a user support device 10 according to the present disclosure. [Figure 2] 2 is a block diagram illustrating an example configuration of a user support system 200 according to the present disclosure. [Figure 3] 3 is a diagram illustrating a first embodiment of the user support system 200 according to the present disclosure. [Figure 4] 4 is a diagram illustrating a second embodiment of the user support system 200 according to the present disclosure. [Figure 5] 5 is a block diagram illustrating an example configuration of a user support device 20 according to the present disclosure. [Figure 6] 6 is a flowchart illustrating an example of a dedicated concierge display operation of the user support system 200 according to the present disclosure. [Figure 7] 7 is a flowchart illustrating a dedicated concierge conversation operation of the user support system 200 according to the present disclosure. [Figure 8] 8 is a block diagram illustrating an example configuration of a computer 500 according to the present disclosure. [Description of Embodiments]

[0013] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant descriptions will be omitted as necessary for clarity of explanation.

[0014] (First Embodiment) First, the configuration of a user support apparatus 10 according to the first embodiment will be described with reference to Figure 1.

[0015] Figure 1 is a block diagram illustrating an example of the configuration of the user support apparatus 10 according to the present disclosure. As shown in Figure 1, the user support apparatus 10 includes an image acquisition unit 11, an identification unit 12, a position acquisition unit 13, an object output unit 14, an instruction acquisition unit 15, and a response output unit 16.

[0016] The user support apparatus 10 supports users who use services. The image acquisition unit 11 of the user support apparatus 10 acquires image information including a user who uses the service. The identification unit 12 identifies the user based on the image information. The position acquisition unit 13 acquires position information of the identified user. The object output unit 14 outputs a control signal for outputting object information of the user's dedicated concierge to a display area corresponding to the user's position information among a plurality of display areas on the user's movement route. The instruction acquisition unit 15 acquires user instruction information including natural language. The response output unit 16 generates a dedicated concierge response based on the user's instruction information using a large language model, and outputs a control signal for executing processing corresponding to the dedicated concierge response to an output apparatus.

[0017] By doing so, the user support apparatus 10 enables the user to experience a scenario in which a dedicated concierge accompanies the user as part of the service. Therefore, the user support apparatus 10 can improve the satisfaction level of users using the service.

[0018] (Second embodiment) Next, the configuration of the user support system 200 according to the second embodiment will be described using Figures 2 to 5.

[0019] The user support system 200 is a system that embodies the user support device 10 according to the first embodiment. Similar to the user support device 10, the user support system 200 assists users of the service. The facilities providing the service are, for example, smart retail environments and restaurants. Here, a smart retail environment refers to a retail store whose operations are streamlined and automated by utilizing technologies such as AI, IoT, and big data.

[0020] Figure 2 is a block diagram showing an example of the configuration of the user support system 200 according to this disclosure. Figure 3 is a diagram showing a first embodiment of the user support system 200 according to this disclosure. Figure 4 is a diagram showing a second embodiment of the user support system 200 according to this disclosure.

[0021] As shown in Figure 2, the user assistance system 200 comprises a user assistance device 20, at least one imaging device 1, at least one display device 2, at least one audio input device 3, and at least one audio output device 4.

[0022] The user support device 20 is, for example, a server. The user support device 20 is a device that performs processing to support users who use the service. The user support device 20 is installed inside or outside the facility that provides the service. The user support device 20 communicates with each of the imaging devices 1, each display device 2, each audio input device 3, and each audio output device 4. The communication method is wireless or wired.

[0023] Each imaging device 1 is a device that captures images, such as a visible light camera. Each imaging device 1 transmits the captured image information and the identification information of the imaging device 1 to the user support device 20.

[0024] Each imaging device 1 is installed in the facility along the user's movement path. Here, the devices are installed so that a user moving along the movement path is always included in the field of view of one of the imaging devices 1. For example, as shown in Figure 3, in the first embodiment, imaging device 1-1 is installed along the user X's movement path. Assume that user X moves sequentially from position P-1, ..., to position Pn (where n is a natural number). Imaging device 1-1 can image the user at any of the positions from P-1 to Pn. Also, as shown in Figure 4, in the second embodiment, imaging devices 1-1 to 1-n are installed along the user X's movement path. Assume that user X moves sequentially from position P-1, ..., to position Pn (where n is a natural number). Imaging devices 1-1 to 1-n can each image the user at position P-1 to the user at position Pn, respectively.

[0025] Returning to the explanation of Figure 2, each display device 2 is a device for visually displaying information, such as a display. Each display device 2 may also be a projector or the like. Each display device 2 has at least one display area 210. Each display area 210 is the area on which information is to be displayed. Here, if a single display device 2 has multiple display areas 210, each of the multiple display areas 210 may have overlapping areas. Each display device 2 receives a control signal from the user assistance device 20 for displaying information and displays the information in the display area 210 designated by the user assistance device 20 based on the control signal.

[0026] Each display device 2 is installed along the user's movement path in the facility providing the service. It is preferable that each display device 2 is installed so that a user moving along the movement path can always see one of the display areas 210. For example, as shown in Figure 3, in the first embodiment, display device 2-1 including display areas 210-1 to 210-n is installed along the user's movement path. Also, as shown in Figure 4, in the second embodiment, display device 2-1 including display area 210-1 to display device 2-n including display area 210-n are installed along the user's movement path. In the first and second embodiments, display areas 210-1 to 210-n are associated with positions P-1 to Pn, respectively, and information is displayed to the user at position P-1 to the user at position Pn.

[0027] Returning to the explanation of Figure 2, each voice input device 3 is a device for receiving voice input, such as a microphone. The voice input device 3 receives instruction information, including natural language, from the user via voice and transmits this instruction information to the user assistance device 20.

[0028] Each voice input device 3 is installed along the user's movement path in the facility providing the service. For example, each voice input device 3 is installed embedded in the wall or ceiling of the facility, behind the display area 210. Each voice input device 3 is installed so that a user moving along the movement path is always included in the voice input range of one of the voice input devices 3. Here, it is preferable that each voice input device 3 is installed at predetermined intervals so that the voice input ranges of each voice input device 3 do not overlap. For example, as shown in Figure 3, in the first embodiment, voice input devices 3-1 to 3-n are installed along the user's movement path. Voice input devices 3-1 to 3-n are associated with positions P-1 to Pn, respectively, and can receive voice input from users at positions P-1 to Pn. The same applies to the second embodiment shown in Figure 4.

[0029] Returning to the explanation of Figure 2, each audio output device 4 is a device for outputting sound, such as a speaker. Each audio output device 4 receives audio information from the user assistance device 20 and outputs that audio information.

[0030] Each audio output device 4 is installed along the user's movement path in the facility providing the service. For example, each audio output device 4 is installed embedded in the wall or ceiling of the facility, behind the display area 210. Here, each audio output device 4 is installed so that a user moving along the movement path is always included in the audio output range of one of the audio output devices 4. Here, it is preferable that each audio output device 4 is installed at a predetermined interval so that the audio output ranges of each audio output device 4 do not overlap. For example, as shown in Figure 3, in the first embodiment, audio output devices 4-1 to 4-n are installed along the user's movement path. Audio output devices 4-1 to 4-n are associated with positions P-1 to Pn, respectively, and can output audio to users at positions P-1 to Pn. The same applies to the second embodiment shown in Figure 4.

[0031] Returning to the explanation of Figure 2, the user assistance system 200 may use an input device for inputting text data, such as a touch panel, instead of the voice input device 3. Similarly, an output device for outputting text data, such as a display, may be used instead of each voice output device 4. The voice input devices 3 and voice output devices 4 may not be installed along the user's movement path within the facility; instead, they may be mounted on the mobile terminal used by each user. Furthermore, an input device capable of accepting input via sign language or body language using a gesture recognition camera or motion sensor may be used. Additionally, an input device capable of reading characters, figures, QR codes (registered trademarks), etc., presented by the user using image recognition technology may be used. Furthermore, to accommodate foreign users, an input device capable of accepting input in various languages ​​may be used by incorporating a multilingual speech recognition system or an automatic translation system. Finally, an output device that allows for more natural user-system interaction using an augmented reality (AR) device may be used.

[0032] Figure 5 is a block diagram showing an example of the configuration of the user assistance device 20 according to this disclosure. As shown in Figure 5, the user assistance device 20 includes a storage unit 21, an image acquisition unit 22, an identification unit 23, a position acquisition unit 24, a dedicated concierge determination unit 25, an object output unit 26, an instruction acquisition unit 27, and a response output unit 28.

[0033] The memory unit 21 stores user information, concierge information, equipment information, area information, and the concierge AI model.

[0034] User information includes user identification information linked to the user's facial information and attribute information. Facial information includes information on facial features. Attribute information includes information on external attributes (e.g., gender, age, and body type) and internal attributes (e.g., hobbies and preferences).

[0035] Concierge information includes concierge identification information, user identification information, and object information linked to the concierge. Object information includes, for example, information about the object's graphics (e.g., character design, clothing, etc.). The object's clothing may take into account seasonal events (New Year's, Christmas, Halloween, etc.) or store events (sales, etc.).

[0036] The equipment information includes information that links the location information of equipment installed in a facility providing services to the identification information of said equipment. Specifically, the equipment information includes information that links the location information of the imaging device 1 to the identification information of said imaging device 1. The equipment information also includes information that links the location information of the display device 2 and the identification information of at least one display area 210 of the display device 2 to the identification information of the display area 210. The equipment information also includes information that links the location information of the display area 210 to the identification information of the display area 210. The equipment information also includes information that links the location information of the voice input device 3 to the identification information of said voice input device 3. The equipment information also includes information that links the location information of the voice output device 4 to the identification information of said voice output device 4.

[0037] Area information includes information that links location information of a facility area to identification information of that area. Here, a facility area refers to, for example, an entrance area, a product display area, an exit area, or a checkout area.

[0038] A concierge AI model is an AI model used for a concierge to converse with a user. A concierge AI model is a large-scale language model (LLM). The concierge AI model has pre-trained functions to be performed. The concierge AI model generates text data representing the concierge's response to user instructions. User instructions may include, at a minimum, natural language, text, audio, video, images, etc.

[0039] The image acquisition unit 22 acquires image information, including the user, and identification information of the imaging device 1 associated with the image information from the imaging device 1. The image acquisition unit 22 acquires this information in chronological order.

[0040] The identification unit 23 identifies the user included in the image information based on the image information. In other words, the identification unit 23 obtains user identification information included in the image information based on the image information. Specifically, the identification unit 23 uses a facial recognition AI model (not shown) to detect the face portion from the image information and extracts facial information that shows the feature quantities of the face portion. Then, the identification unit 23 obtains user identification information linked to the facial information corresponding to the extracted facial information from the user information in the storage unit 21.

[0041] The position acquisition unit 24 acquires the user's location information based on the image information and the identification information of the imaging device 1 associated with the image information. The location information is information about the position in a three-dimensional coordinate system. Specifically, the position acquisition unit 24 acquires the location information of the imaging device 1, which is associated with the identification information of the imaging device 1, from the device information of the storage unit 21. Then, the position acquisition unit 24 acquires the user's location information based on the location information of the imaging device 1 and the user's position in the image information.

[0042] The dedicated concierge determination unit 25 acquires the identification information of the user's dedicated concierge based on the user's identification information. Specifically, the dedicated concierge determination unit 25 acquires the identification information of the concierge corresponding to the user's identification information from the concierge information in the storage unit 21 as the identification information of the dedicated concierge.

[0043] The object output unit 26 obtains object information for the dedicated concierge based on the identification information of the dedicated concierge. Specifically, the object output unit 26 obtains object information corresponding to the identification information of the dedicated concierge from the concierge information in the storage unit 21.

[0044] The object output unit 26 then outputs a control signal to display the object information of the dedicated concierge in the display area 210 corresponding to the user's location information, among at least one of the display areas 210. Specifically, the object output unit 26 obtains identification information of the display area 210 associated with the location information corresponding to the user's location information from the device information of the storage unit 21. The object output unit 26 outputs a control signal to the display device 2 equipped with the display area 210 to display the object information of the dedicated concierge in the display area 210 from which the identification information has been obtained. For example, the display area 210 includes the wall of a store, the door of a glass case containing beverages or frozen foods in a store such as a convenience store, or the curtain for a refrigerated display case on a shelf where fresh food is placed in a supermarket, etc. For example, the object output unit 26 selects the optimal display area 210 based on the user's location information, taking into account the distance from the user and the angle of their line of sight. For example, the object output unit 26 selects a small display area 210 if the user is close, and a large display area 210 if the user is far away. Furthermore, the object output unit 26 dynamically adjusts the display size and orientation of the concierge object information according to the user's viewing angle, thereby ensuring optimal visibility at all times.

[0045] As shown in Figure 3, in the first embodiment, when the user is at position P-1, the object output unit 26 outputs a control signal to the display device 2-1 to display object information of the dedicated concierge Y in the display area 210-1 corresponding to position P-1. Furthermore, when user X moves from position P-1 to position Pn, the object output unit 26 outputs a control signal to the display device 2-1 to display object information of the dedicated concierge Y in the display area 210-n corresponding to position Pn. Furthermore, as shown in Figure 4, in the second embodiment, when user X is at position P-1, the object output unit 26 outputs a control signal to the display device 2-1 to display object information of the dedicated concierge Y in the display area 210-1 corresponding to user X's position P-1. Furthermore, when user X moves from position P-1 to position Pn, the object output unit 26 outputs a control signal to the display device 2-n to display object information of the dedicated concierge Y in the display area 210-n corresponding to user X's position Pn.

[0046] Returning to the explanation of Figure 5, the instruction acquisition unit 27 acquires user instruction information, including natural language, from the voice input device 3 corresponding to the user's location information, which is one of at least one voice input device 3 installed along the user's movement path. Specifically, the instruction acquisition unit 27 acquires identification information of the voice input device 3 linked to the location information corresponding to the user's location information from the device information of the storage unit 21. The instruction acquisition unit 27 acquires user instruction information, including natural language, from the voice input device 3 from which the identification information has been acquired. Then, the instruction acquisition unit 27 converts the user instruction information from voice information to text information.

[0047] The response output unit 28 acquires identification information for the area containing the user's location information. The response output unit 28 uses a concierge AI model to generate a response from the dedicated concierge based on the user's instruction information and the area identification information. The response output unit 28 then outputs a control signal to the voice output device 4 to execute the processing corresponding to the dedicated concierge's response. Specifically, the response output unit 28 acquires identification information for the area associated with the location information corresponding to the user's location information from the area information in the storage unit 21. The response output unit 28 uses a concierge AI model to generate a response from the dedicated concierge to the user's instruction information based on the dedicated concierge's identification information, the user's instruction information, and the area identification information. The response output unit 28 then converts the dedicated concierge's response from text data to voice information. The response output unit 28 acquires identification information for the voice output device 4 associated with the location information corresponding to the user's location information from the device information in the storage unit 21. The response output unit 28 outputs a control signal to the voice output device 4, which has acquired the identification information, causing the voice output device 4 to output the response of the dedicated concierge.

[0048] As shown in Figure 3, in the first embodiment, when user X is at position P-1, the instruction acquisition unit 27 acquires user instruction information from the voice input device 3-1 corresponding to position P-1. Subsequently, the response output unit 28 outputs a control signal to the voice output device 4-1 corresponding to position P-1, causing it to output the dedicated concierge's response to the user's instruction information. Furthermore, when user X moves from position P-1 to position Pn, user instruction information is acquired from the voice input device 3-n corresponding to position Pn. Subsequently, the response output unit 28 outputs a control signal to the voice output device 4-n corresponding to position Pn, causing it to output the dedicated concierge's response to the user's instruction information. The same applies to the second embodiment shown in Figure 4.

[0049] Next, using Figure 6, the dedicated concierge display operation of the user support system 200 according to the second embodiment will be explained.

[0050] Figure 6 is a flowchart showing an example of the dedicated concierge display operation of the user support system 200 related to this disclosure.

[0051] In the following example, the user support system 200 repeatedly performs the operations described in steps S101 to S106 above.

[0052] First, in the user support system 200, a predetermined imaging device 1, installed along the user's travel route, captures an image including the user. The imaging device 1 transmits the captured image information and the identification information of the imaging device 1 associated with the image information to the user support system 20.

[0053] As shown in Figure 6, in step S101, the image acquisition unit 22 of the user support device 20 acquires image information including the user and identification information of the imaging device 1 associated with the image information from the imaging device 1.

[0054] In step S102, the identification unit 23 acquires user identification information based on the image information. In step S103, the position acquisition unit 24 acquires the position information of the user whose identification information has been acquired, based on the image information and the identification information of the imaging device 1 associated with the image information.

[0055] In step S104, the dedicated concierge determination unit 25 obtains the identification information of the user's dedicated concierge based on the user's identification information. In step S105, the dedicated concierge determination unit 25 obtains the object information of the dedicated concierge based on the identification information of the dedicated concierge.

[0056] In step S106, the object output unit 26 outputs a control signal to the display device 2 equipped with at least one display area 210, which corresponds to the user's location information, to display the object information of the dedicated concierge. The display device 2 equipped with the display area 210 then displays the object information of the dedicated concierge based on the control signal.

[0057] By doing so, the user support system 200 allows users to experience what it's like to have a dedicated concierge accompanying them as part of the service. Therefore, the user support system 200 can improve user satisfaction with the service.

[0058] Next, using Figure 7, the dedicated concierge conversation operation of the user support system 200 according to the second embodiment will be explained.

[0059] Figure 7 is a flowchart showing an example of a dedicated concierge conversation operation in the user support device 20 of the user support system 200 related to this disclosure.

[0060] As shown in Figure 7, in step S201, the instruction acquisition unit 27 of the user support device 20 acquires the user's identification information, the user's location information, and the identification information of the dedicated concierge. This information is the same information acquired in steps S101 to S104 described above.

[0061] Next, in step S202, the instruction acquisition unit 27 acquires user instruction information, including natural language, from one of the at least one voice input devices 3 installed along the user's movement path, specifically the voice input device 3 corresponding to the user's location information. Here, the acquired user instruction information is converted from voice information to text information.

[0062] Next, in step S203, the response output unit 28 acquires area identification information, including the user's location information.

[0063] Next, in step S204, the response output unit 28 uses the concierge AI model to generate a response from the dedicated concierge based on the dedicated concierge's identification information, the user's instruction information, and the area information. Here, the generated response from the dedicated concierge is converted from text information to voice information.

[0064] Next, in step S205, the response output unit 28 outputs an audio output device 4 that corresponds to the user's location information from among the at least one audio output device 4 installed along the user's movement path. A control signal is output to the audio output device 4 to cause it to perform processing in accordance with the response of the dedicated concierge. The audio output device 4 then outputs the response of the dedicated concierge based on the control signal.

[0065] By doing so, the user support system 200 allows users to constantly communicate with their dedicated concierge. Therefore, users can always receive support from their dedicated concierge if they have any questions about the service. Consequently, the user support system 200 can improve user satisfaction with the service. Furthermore, with the user support system 200, users can receive appropriate support from a dedicated concierge, tailored to the specific area of ​​the facility. Therefore, the user support system 200 can improve user satisfaction with the service.

[0066] (Modified version of the second embodiment) Furthermore, the configuration of the user support system 200 according to the second embodiment can also be modified to the following configuration. Next, a modified example of the user support system 200 according to the second embodiment will be described.

[0067] In the user support system 200 according to the above embodiment, the concierge AI model stored in the memory unit 21 of the user support device 20 generates the concierge's response to the user's instruction information as text data from the user's instruction information, including natural language. The response output unit 28 uses the concierge AI model to generate the dedicated concierge's response as text data based on the dedicated concierge's identification information, the user's instruction information, and the area's identification information. In a modified example, the concierge AI model includes a function to further generate motion data, such as gestures and movements of the concierge's object, as the concierge's response, based on the text data generated as the concierge's response. The response output unit 28 then uses the concierge AI model to generate motion data including the dedicated concierge's response to the user's instruction information, based on the dedicated concierge's identification information, the user's instruction information, the area information's identification information, and the dedicated concierge's object information. The response output unit 28 then outputs a control signal to the display area 210, where the object information of the dedicated concierge is displayed, to display the dedicated concierge's response as motion. This makes it easier for the user to understand the content of the service. It also makes it easier for the user to feel a sense of familiarity with the dedicated concierge. Therefore, the user support system 200 can improve the satisfaction of users using the service.

[0068] Furthermore, in the user support system 200 according to the above embodiment, the response output unit 28 of the user support device 20 uses a concierge AI model to generate a response from the dedicated concierge based on the identification information of the dedicated concierge, the user's instruction information, and the area identification information.

[0069] In the modified version, the user information in the memory unit 21 includes information that links the user's identification information with the user's facial information and attribute information, as well as the response history information of the dedicated concierge. The dedicated concierge's response history information is a history of the dedicated concierge's past responses in text data. The response output unit 28 obtains the dedicated concierge's response history information corresponding to the user's identification information from the user information in the memory unit 21. The response output unit 28 generates the dedicated concierge's response based on the dedicated concierge's identification information, the user's instruction information, the area information identification information, and the response history information. For example, the response output unit 28 includes the content of past conversations with the user in the dedicated concierge's response. Furthermore, the response output unit 28 generates the dedicated concierge's response in a way that increases the level of familiarity with the user according to the content of past conversations with the user. By doing so, the user can feel a greater sense of fit with the dedicated concierge, or in other words, a greater sense of becoming a dedicated concierge. Therefore, the user support system 200 can improve the satisfaction of users who use the service.

[0070] In other variations, the user information in the memory unit 21 includes user identification information, as well as information linked to the user's facial information, attribute information, and service usage history information. The service usage history information includes, for example, e-commerce (EC) history and real-world history. The response output unit 28 obtains the user's service usage history information corresponding to the user's identification information from the user information in the memory unit 21. The response output unit 28 generates a response from the dedicated concierge based on the dedicated concierge's identification information, the user's instruction information, the area information identification information, and the service usage history information. For example, the response output unit 28 analyzes the user's preferred products from the service usage history information and generates a response from the dedicated concierge that suggests products the user prefers. By doing so, in the user support system 200, users can receive more appropriate support from the dedicated concierge in the service. Therefore, the user support system 200 can improve user satisfaction with the service.

[0071] In other variations, the memory unit 21 may also store provider request information. This provider request information includes information about the service provider's requests. For example, the service provider's request information might be something like "I want to sell product XX." The response output unit 28 retrieves the provider request information corresponding to the user's identification information from the user information in the memory unit 21. Based on the provider request information, in addition to the dedicated concierge's identification information, the user's instruction information, and the area information identification information, the response output unit 28 generates a response from the dedicated concierge. For example, the response output unit 28 generates a response from the dedicated concierge to suggest products that the service provider wants to sell. By doing so, the user support system 200 can improve not only the user's satisfaction but also the service provider's satisfaction.

[0072] In the user assistance system 200 according to the above embodiment, the object output unit 26 of the user assistance device 20 outputs a control signal to display object information of the dedicated concierge in the display area 210 corresponding to the user's location information, which is one of at least one display area 210. In a modified example, the user assistance device 20 further includes an emotion detection unit (not shown) that detects the user's emotions based on image information. The user's emotions may be, for example, joy, sadness, anger, or confusion. Depending on the detected user emotion, the object output unit 26 outputs a control signal to stop the output of the dedicated concierge's object information or a control signal to output the dedicated concierge's object information in the display area 210 where the dedicated concierge's object information is displayed. Specifically, if the user's emotion is a first emotion (for example, anger), the object output unit 26 outputs a control signal to stop the output of the dedicated concierge's object information in the display area 210 where the dedicated concierge's object information is displayed. Subsequently, if the user's emotion changes from a first emotion (e.g., anger) to a second emotion (e.g., joy) that is different from the first emotion, the object output unit 26 outputs a control signal to the display area 210 where the object information of the personal concierge is displayed, causing the personal concierge's object information to be output. By doing so, the user support system 200 allows the user to feel that the personal concierge is there when they want them to be. Therefore, the user support system 200 can improve the satisfaction of users using the service.

[0073] Furthermore, in the above-described embodiment, the concierge information stored in the storage unit 21 of the user support device 20 includes information that links the concierge's identification information with the user's identification information and object information that the concierge is exclusively assigned to. The exclusive concierge determination unit 25 then obtains the concierge's identification information corresponding to the user's identification information from the concierge information in the storage unit 21 as the exclusive concierge's identification information. In a modified example, the concierge information stored in the storage unit 21 includes information that links the concierge's identification information with the user's identification information and object information that the concierge is exclusively assigned to, in addition to attribute information. The attribute information includes information such as external attributes (e.g., gender, age, and body type) and internal attributes (e.g., hobbies and preferences). The exclusive concierge determination unit 25 obtains the user's attribute information based on the user's identification information, and obtains the user's exclusive concierge's identification information based on the user's attribute information. Specifically, the dedicated concierge selection unit 25 obtains user attribute information linked to the user's identification information from the user information in the memory unit 21. The dedicated concierge selection unit 25 uses a matching AI model (not shown) to obtain the identification information of a concierge linked to the attribute information that matches the user's attribute information from the concierge information in the memory unit 21, and uses the obtained information as the identification information of the dedicated concierge. For example, a machine learning algorithm is used in the matching AI model. For example, the matching AI model has the task of selecting a concierge with a female appearance for a male user. The matching AI model may also select a concierge based on the user's explicit settings. Specifically, the model may select a concierge based on preferences set in advance by the user (e.g., concierge's appearance, personality, expertise, etc.). The matching AI model may also select a concierge with appropriate expertise and experience depending on the type of service or information the user is seeking.

[0074] In the user assistance system 200 according to the above embodiment, the object output unit 26 of the user assistance device 20 outputs a control signal to a display device 2 equipped with at least one display area 210, which corresponds to the user's location information, to display object information of the dedicated concierge in the display area 210 corresponding to the user's location information. Each display device 2 is a display or projector. In a modified example, each display device 2 is a holographic device for displaying 3D holograms. In that case, the displayed object information becomes 3D hologram information. Also, the display area 210 becomes a three-dimensional space on which the 3D hologram is displayed.

[0075] (An example of a response from a dedicated concierge according to the second embodiment) Next, an example of a response from a dedicated concierge in the user support system 200 according to the second embodiment will be described.

[0076] <Smart Retail Environment> In a smart retail environment, the responses of a dedicated concierge are generated by a dedicated concierge AI model based on the following policies for each area where the user is located.

[0077] (1-1) Entrance area • Greetings and guidance to customers who visit the store. • Display of store map: • Sharing store information (e.g., "We're having a sale in the XX area. The YY area is currently very crowded.") • Recommendations for areas where recommended products or products of user interest are sold. (1-2) Exit area • Farewell and expression of gratitude • Recommendations for the next store to visit (1-3) Settlement Area • Support for smooth accounting processing • Payment method explanation • Information about our point card • Providing valuable coupon information • Integrates with the order system to execute at least one of the following: order placement and order fulfillment. (1-4) Product display area • Directions to the location of the product • Product features and usage instructions • Check stock availability • Recommended products • Recommendations based on your previous purchase history • In the clothing section, they recommend, "These bottoms go well with the top you bought the other day." • In the food section, they recommend products like, "You've been consuming too much salt lately, so this product might be better for you." (1-5) Fitting room area • Support for trying on clothes • Support for size and color exchanges • Styling advice (1-6) Waiting area • Waiting time information • Providing store information (1-7) Customer Service Area • Inquiry handling • Claim handling • Information on various services (1-8) Product Pickup Area • Confirmation and handover of ordered items • Check the packaging condition of the product. • Simple instructions on how to use it (1-9) Return / Exchange Area • Instructions for returns and exchanges • Support for refund processing • Suggestion of alternative products (1-10) Sale / Special Offer Area • Detailed explanation of sale information • Check stock availability • Suggestions for similar products (1-11) New Product Exhibition Area • Description of the new product's features ·demonstration • Reservations accepted (1-12) Seasonal Products Area • Product suggestions tailored to the season • Instructions on how to use and care for the product. • Coordination proposals (1-13) Passageway area • Directions to your destination • Product location guide • Guidance during crowded times (1-14) Escalator / Elevator Area • Support for safe use • Hierarchical Guide (1-15) Toilet area • Directions • Information on available toilets • Information on available restrooms in nearby facilities or within the same facility. (1-16) Parking and bicycle parking area • Information on available spaces • Explanation of parking and bicycle parking methods • Information on the pricing system (1-17) Product Information / Information Board Area • Providing detailed information • Support for product comparison • Suggestions for related products (1-18) Gift wrapping area • Explanation of wrapping options • Offering a gift wrapping service • Support for filling out message cards (1-19) Event Area • Explanation of the event Instructions on how to participate • Introduction of related products (1-20) Kids Area • Instructions on how to play • Information for parents / guardians (1-21) Online order pickup area • Order confirmation and handover • Product verification support • Return and exchange information

[0078] <Restaurant> In restaurants, the responses from the dedicated concierge are generated by a dedicated concierge AI model based on the following policies for each area where the user is located.

[0079] (2-1) Entrance area • Welcoming and greeting customers who visit the store. • Explanation of the store's congestion status • Confirmation of reservation (2-2) Exit Area • Seeing them off and expressing gratitude • Taxi calling support • Calling out to check for forgotten items (2-3) Waiting area • Guidance and management of waiting times • Menu explanation beforehand • Pre-orders accepted (2-4) Reception Area • Smooth guidance and seating arrangements • Handling special requests • Seat requests • Coordination of large groups (2-5) Main Dining Area • Comfortable seating arrangements • Menu description and recommendations • Confirmation of specific dietary requirements or restrictions • Allergy check (2-6) Private Room Area • Ensuring privacy • Explanation of the facilities in the private room · Explanation of the call button for the private room • Proposal of special services • Proposal for anniversary service (2-7) Terrace seating area • Guidance based on weather conditions • For example, suggestions for sunshades and heaters. • Explanation of scenic points • Suggestions for insect repellent measures (2-8) Bar area • Cocktail and wine recommendations • Support for conversation with the bartender • Food pairing suggestions (2-9) Counter seating area • Explanation of the cooking process • Support for conversations with the chef • Information about the special menu (2-10) Settlement Area • Payment method information • Explanation of receipts and statements • Checking the application of point cards and discounts. (2-11) Toilet area • Directions to the location • Description of amenities (2-12) Smoking area • Explanation of smoking rules • Guidance for non-smokers (2-13) Drink Station Area • Description of the types and characteristics of beverages • Information on cup sizes and straws • Explanation of the refill system (2-14) Salad bar area • Description of fresh ingredients • Introduction to the types and characteristics of dressings • Providing allergy information (2-15) Dessert Area • Introducing today's recommended dessert • Providing information on calories and ingredients • Dessert and drink pairing suggestions (2-16) Takeout Area • Explanation of the takeout menu • Advice on packaging and storage methods • Information on the reservation system (2-17) Emergency Exit Area • Explanation of emergency evacuation routes • Guidance on the location and use of safety equipment • Explanation of procedures for responding to disasters (2-18) Elevator / Staircase Area • Floor guide and description of the features of each floor • Barrier-free access information • Instructions on how to use the elevator (2-19) Orderable areas • Support during ordering • Support, especially when ordering via touch panel (for example, elderly people often have difficulty operating touch panels, so ordering on their behalf). The specific support provided is as follows: 1. Order support via voice recognition: The system recognizes the user's voice commands, and a dedicated concierge enters the order on their behalf. 2. Zoom function: The text and buttons on the touch panel are automatically enlarged, making them easier to see for users with presbyopia. 3. Contrast adjustment function: It automatically adjusts the screen contrast to improve visibility. 4. Simple operation mode: It simplifies complex operations and supports user interaction with large buttons and a concise display. 5. Step-by-step guidance: A dedicated concierge will visually demonstrate each step on the screen while also providing audio explanations. 6. Proxy input option: A dedicated concierge will handle all order entry on behalf of the user, according to their requests. 7. Order confirmation and correction support: The system reads out the entered order details clearly and provides voice instructions to easily make any necessary corrections. 8. Multimodal input support: It offers multiple input methods, such as touch, voice, and gestures, to suit the user's preferences and abilities. 9. Remember personalized settings: It remembers the user's past usage patterns and settings, and automatically provides the optimal display and operating environment on the next visit.

[0080] The above describes an example of a response from a dedicated concierge in the user support system 200, but this is not limited to the example described above and can be modified as appropriate depending on the service.

[0081] Next, the hardware configuration of the user support device 10 and user support device 20 of the user support system 200 according to the second embodiment will be described.

[0082] Each functional component of the user assistance device 10 and the user assistance device 20 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program to control it). The case in which each functional component of the above device is implemented by a combination of hardware and software will be further explained below.

[0083] Figure 8 is a block diagram illustrating the hardware configuration of the computer 500 according to this disclosure. Both the user support device 10 and the user support device 20 can be implemented with the computer 500 having the hardware configuration shown in Figure 8. The computer 500 may be a portable computer such as a smartphone or tablet terminal, or a stationary computer such as a PC (personal computer). The computer 500 may be a dedicated computer or a general-purpose computer.

[0084] For example, by installing a predetermined application on the computer 500, the computer 500 can be given the desired functions. By installing applications that realize the respective functions of the user support device 10 and the user support device 20 on the computer 500, each function is realized on the computer 500.

[0085] Computer 500 includes a bus 501, a processor 502, memory 503, a storage device 504, an input / output interface (I / F) 505, and a network interface (I / F) 506. Bus 501 is a data transmission path for the processor 502, memory 503, storage device 504, input / output interface 505, and network interface 506 to send and receive data to and from each other. However, the method of connecting the processor 502 and other components to each other is not limited to bus connection.

[0086] The processor 502 is a variety of processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 503 is the main memory, implemented using RAM (Random Access Memory), etc. The storage device 504 is the auxiliary storage, implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0087] The input / output interface 505 is an interface for connecting the computer 500 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 505.

[0088] The network interface 506 is an interface for connecting computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0089] The storage device 504 stores programs for realizing desired functions. For example, the storage device 504 of computer 500 stores programs for realizing the functions of user assistance device 10 and user assistance device 20. The processor 502 reads these programs into memory 503 and executes them to realize each function.

[0090] These programs, when loaded into a computer, include a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The programs may be stored on non-temporary computer-readable media or tangible storage media. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs), or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray® discs, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. The programs may be transmitted over temporary computer-readable media or communication media. Examples, but not limited to, include electrical, optical, acoustic, or other forms of propagating signals.

[0091] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.

[0092] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) An image acquisition unit that acquires image information including the user of the service, An identification unit that identifies the user based on the image information, A location acquisition unit that acquires the location information of the identified user, An object output unit outputs a control signal to output object information of the user's personal concierge to a display area linked to the location information corresponding to the user's location information, among a plurality of display areas along the user's movement path. An instruction acquisition unit that acquires the user's instruction information, The system includes a response output unit that uses a large-scale language model to generate a response from the dedicated concierge based on the user's instruction information, and outputs a control signal to an output device to execute processing corresponding to the dedicated concierge's response. User assistance device. (Note 2) The response output unit is, The identification information of the area containing the user's location information is obtained, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the area identification information. User assistance device as described in Appendix 1. (Note 3) The response output unit is, Obtain the historical information of the responses of the aforementioned dedicated concierge, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the history information of the dedicated concierge's responses. User assistance device as described in Appendix 1. (Note 4) The system further includes an emotion detection unit that detects the user's emotions based on the aforementioned image information, The object output unit is, Depending on the user's emotions, a control signal to stop the output of the dedicated concierge's object information or a control signal to output the dedicated concierge's object information will be output to the display area where the object information of the dedicated concierge is displayed. User assistance device as described in Appendix 1. (Note 5) The response output unit is, The user's service usage history information is obtained, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the user's service usage history information. User assistance device as described in Appendix 1. (Note 6) The response output unit is, Obtain the request information of the service provider, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the service provider's request information. User assistance device as described in Appendix 1. (Note 7) The user's instruction information includes at least one of natural language, images, audio, and video. User assistance device as described in Appendix 1. (Note 8) Computers Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, The location information of the identified user is obtained, A control signal is output to the display area linked to the location information corresponding to the user's location information among the multiple display areas along the user's movement path, causing the object information of the user's dedicated concierge to be displayed. The user's instruction information is obtained, Using a large-scale language model, the system generates a response from the dedicated concierge based on the user's instruction information, and outputs a control signal to an output device to execute the processing corresponding to the dedicated concierge's response. User support methods. (Note 9) Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, The location information of the identified user is obtained, A control signal is output to the display area linked to the location information corresponding to the user's location information among the multiple display areas along the user's movement path, causing the object information of the user's dedicated concierge to be displayed. The user's instruction information is obtained, Using a large-scale language model, the system generates a response from the dedicated concierge based on the user's instructions, and then causes the computer to output a control signal to an output device to execute a process corresponding to the dedicated concierge's response. program.

[0093] Some or all of the elements (e.g., configuration and function) described in Appendices 2 through 7 that are dependent on Appendice 1 may also be dependent on Appendices 8 and 9, respectively, in the same manner as those described in Appendices 2 through 7. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software. [Explanation of Symbols]

[0094] 1. 1-1 to 1-n imaging device 2, 2-1~2-n display device 3. 3-1~3-n Voice Input Device 4. 4-1~4-n Audio output device 10, 20 User assistance devices 11, 22 Image acquisition unit 12, 23 Identification section 13, 24 Position acquisition part 14, 26 Object Output Section 15, 27 Instruction acquisition part 16, 28 Response Output Section 21 Memory section 25 Exclusive Concierge Selection Department 200 User Support Systems 210, 210-1~210-n display area 500 Computers 501 Bus 502 Processors 503 memory 504 Storage Device 505 Input / Output Interface (I / F) 505 Input / Output Interface 506 Network Interface (I / F) 506 Network Interface

Claims

1. An image acquisition unit that acquires image information including the user of the service, An identification unit that identifies the user based on the image information, A location acquisition unit that acquires the location information of the identified user, An object output unit outputs a control signal to output object information of the user's personal concierge to a display area linked to the location information corresponding to the user's location information, among a plurality of display areas along the user's movement path. An instruction acquisition unit that acquires the user's instruction information, The system includes a response output unit that uses a large-scale language model to generate a response from the dedicated concierge based on the user's instruction information, and outputs a control signal to an output device to execute processing corresponding to the dedicated concierge's response. User assistance device.

2. The response output unit is, The identification information of the area containing the user's location information is obtained, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the area identification information. The user support device according to claim 1.

3. The response output unit is, Obtain the historical information of the responses of the aforementioned dedicated concierge, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the history information of the dedicated concierge's responses. The user support device according to claim 1.

4. The system further includes an emotion detection unit that detects the user's emotions based on the aforementioned image information, The object output unit is, Depending on the user's emotions, a control signal to stop the output of the dedicated concierge's object information or a control signal to output the dedicated concierge's object information will be output to the display area where the object information of the dedicated concierge is displayed. The user support device according to claim 1.

5. The response output unit is, The user's service usage history information is obtained, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the user's service usage history information. The user support device according to claim 1.

6. The response output unit is, Obtain the request information of the service provider, Using the aforementioned large-scale language model, the response of the dedicated concierge is generated based on the user's instruction information and the service provider's request information. The user support device according to claim 1.

7. The user's instruction information includes at least one of natural language, images, audio, and video. The user support device according to claim 1.

8. Computers Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, The location information of the identified user is obtained, A control signal is output to the display area linked to the location information corresponding to the user's location information among the multiple display areas along the user's movement path, causing the object information of the user's dedicated concierge to be displayed. The user's instruction information is obtained, Using a large-scale language model, the system generates a response from the dedicated concierge based on the user's instruction information, and outputs a control signal to an output device to execute the processing corresponding to the dedicated concierge's response. User support methods.

9. Acquire image information including the user using the service. Based on the aforementioned image information, the user is identified, The location information of the identified user is obtained, A control signal is output to the display area linked to the location information corresponding to the user's location information among the multiple display areas along the user's movement path, causing the object information of the user's dedicated concierge to be displayed. The user's instruction information is obtained, Using a large-scale language model, the system generates a response from the dedicated concierge based on the user's instructions, and then causes the computer to output a control signal to an output device to execute a process corresponding to the dedicated concierge's response. program.

Images