Customer service support device and program

Abstract

To provide a sales promotion support device and program that enables efficient customer service support. [Solution] The customer service support device of the embodiment is a customer service support device that serves as a point of contact with the customer, and comprises: a first acquisition means for acquiring captured images from an imaging device that images the area around the customer service support device; a second acquisition means for acquiring sensing results from a sensor device that senses information related to the operation of the customer service support device; an estimation means for estimating the state of the customer service support device and the state of the area around the customer service support device based on the captured images and the sensing results; a third acquisition means for acquiring a response sentence generated by a generation device that has a generation AI function that generates a response sentence for customer service according to the input state conditions, by inputting the estimation results of the estimation means; and an output means for outputting the response sentence.

Description

Technical Field

[0001] Embodiments of the present invention relate to a customer service support device and a program.

Background Art

[0002] Conventionally, in a store, as part of customer service support, a device capable of providing voice guidance or the like (hereinafter also referred to as a customer service support device) may be used. For example, as a customer service support device, a customer service robot that provides voice guidance may be used. Conventionally, a technique has been proposed in which questions are presented to a respondent from a robot, and products are recommended based on the answers obtained from the respondent.

[0003] However, in conventional customer service support devices, the output voice guidance is stereotyped, and the same voice guidance is mechanically output even when the usage scenarios for customer service are different. Therefore, customers may become indifferent to the voice guidance issued by the customer service support device, and there is room for further improvement from the perspective of customer service.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a customer service support device and a program capable of efficiently performing customer service support.

Means for Solving the Problems

[0005] The customer service support device of the embodiment is a customer service support device that serves as a point of contact with the customer, and comprises: a first acquisition means for acquiring captured images from an imaging device that images the area around the customer service support device; a second acquisition means for acquiring sensing results from a sensor device that senses information related to the operation of the customer service support device; an estimation means for estimating the state of the customer service support device and the state of the area around the customer service support device based on the captured images and the sensing results; a third acquisition means for acquiring a response sentence generated by a generation device that has a generation AI function that generates a response sentence for customer service according to the input state conditions, by inputting the estimation results of the estimation means; and an output means for outputting the response sentence. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of the system configuration of the customer service support system according to the embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of a customer service robot according to the embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of a server device according to this embodiment. [Figure 4] Figure 4 shows an example of the functional configuration of a customer service robot and server device according to an embodiment. [Figure 5] Figure 5 is a sequence diagram showing an example of the processing performed by the customer service robot and server device of the embodiment. [Figure 6] Figure 6 is a flowchart showing an example of the processing performed by the customer service robot of the embodiment. [Modes for carrying out the invention]

[0007] The customer service support device and program according to the embodiment will be described below with reference to the drawings. The embodiment described below describes an example applied to an autonomous customer service robot used in stores such as restaurants, but is not limited to this embodiment.

[0008] Figure 1 shows an example of the system configuration of a customer service support system according to an embodiment. The customer service support system 1 includes a customer service robot 10 and a server device 30. The customer service robot 10 and the server device 30 are connected to each other via a network N so as to be able to communicate.

[0009] Network N can utilize the internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc., individually or in appropriate combinations. In this embodiment, a LAN installed within a store is used as an example of network N. The number of customer service robots 10 in the customer service support system 1 is not limited to the illustrated example.

[0010] The customer service robot 10 is an example of a customer service support device that serves as a point of contact (touch point) with customers. One or more customer service robots 10 are installed in a store. The customer service robot 10 autonomously moves around the store and provides various customer service support such as attracting customers, guiding them to their seats, and delivering (serving) products. The customer service robot 10 is equipped with a mobile unit 21 for moving the robot 10, as well as shelves for placing products.

[0011] The customer service robot 10 estimates the state of its own device and the state of its surroundings based on information obtained through input devices such as the imaging unit 22 and the sensor unit 27, which will be described later. The customer service robot 10 generates conditional information using the estimated state of its own device and the state of its surroundings as conditions for generating a customer service message (response text) and transmits it to the server device 30. When the customer service robot 10 receives the response text from the server device 30, it outputs the response text via output devices such as the display unit 23 and the voice output unit 26, which will be described later.

[0012] The server device 30 is an example of a generation device having a generation AI function. The server device 30 is a server device installed, for example, in the back room of a store. The server device 30 has a generation AI (Artificial Intelligence) function specialized for natural language processing. When the server device 30 receives condition information from the customer service robot 10, it generates a prompt based on that condition information. Then, the server device 30 generates a response sentence corresponding to the prompt using its generation AI function and sends the generated response sentence to the customer service robot 10.

[0013] In Figure 1, the server device 30 is shown as a single device, but in practice, the server device 30 may be realized through the cooperation of multiple devices. Furthermore, the server device 30 may be a cloud server (or cloud system) realized by multiple network-connected devices. The system configuration between the customer service robot 10 and the server device 30 can also be described as a client-server system, an edge device-edge computer system, or the like.

[0014] Next, the configuration of the customer service robot 10 and server device 30 described above will be explained.

[0015] Figure 2 shows an example of the hardware configuration of a customer service robot 10 according to an embodiment. As shown in Figure 2, the customer service robot 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, and a storage unit 15.

[0016] The CPU 11 is an example of a processor that comprehensively controls all parts of the customer service robot 10. The ROM 12 stores various programs. The RAM 13 is a workspace for displaying programs and various data.

[0017] The communication unit 14 is a wireless communication interface that can be connected to the network N. The communication unit 14 communicates with external devices such as the server device 30 via the network N.

[0018] The storage unit 15 has a storage medium such as a HDD (Hard Disk Drive) or a flash memory, and maintains the stored content even when the power is turned off. The storage unit 15 stores a program 151 that can be executed by the CPU 11 and various setting information. For example, the program 151 includes a program for realizing the functional configuration described later. In addition, the storage unit 15 stores various information related to customer service operations, such as map data indicating a map inside the store.

[0019] The CPU 11, ROM 12, and RAM 13 are connected via a bus or the like to constitute a control unit 100 having a computer configuration. In the control unit 100, the CPU 11 operates according to a program stored in the ROM 12 or the storage unit 15 and expanded in the RAM 13, thereby executing various processes.

[0020] In addition, the customer service robot 10 includes a moving unit 21, an imaging unit 22, a display unit 23, an operation unit 24, an audio input unit 25, an audio output unit 26, and a sensor unit 27, etc.

[0021] The moving unit 21 is provided below the customer service robot 10 and supports the main body of the customer service robot 10 from below. The moving unit 21 includes a driving unit having a motor and a rotation mechanism, etc. In addition, on the bottom surface side of the moving unit 21, a plurality of driving wheels, auxiliary wheels, etc. that are rotationally driven by the driving unit are provided. The driving unit moves (runs) the customer service robot 10 by rotationally driving the driving wheels, auxiliary wheels, etc. under the control of the control unit 100. Specifically, the driving unit runs the customer service robot 10 in an arbitrary direction and at an arbitrary speed by changing the driving driving wheels, the rotational speed of the driving source, and the rotational direction.

[0022] The imaging unit 22 is an imaging device having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary MOS). The imaging unit 22 images the periphery of the customer service robot 10 in order to capture customers existing around the customer service robot 10 or obstacles in its traveling direction. The imaging unit 22 is provided on the front side and the back side of the customer service robot 10, but is not limited thereto and may be provided on only one surface. Note that the front side of the customer service robot 10 means the direction side where the customer service robot 10 faces the customer during normal operation or the traveling direction side when traveling forward. Also, the back side of the customer service robot 10 means the back side of the customer service robot 10 facing the front side.

[0023] The display unit 23 is composed of a display device such as an LCD (Liquid Crystal Display). The display unit 23 displays various information under the control of the control unit 100. The display unit 23 is provided, for example, on both or one of the front side and the back side of the customer service robot 10.

[0024] The operation unit 24 is an input device such as a touch panel provided on the display unit 23. The operation unit 24 outputs the operation content input from a user (for example, a store clerk, etc.) to the CPU 11.

[0025] The voice input unit 25 is a voice input device such as a microphone. The voice input unit 25 picks up the voice around the customer service robot 10. The voice output unit 26 is a voice output device such as a speaker. The voice output unit 26 outputs the voice and sound of the customer service robot 10.

[0026] The sensor unit 27 is various sensor devices capable of sensing various information related to the operation of the customer service robot 10. The sensing result of the sensor unit 27 is output to the control unit 100.

[0027] For example, the sensor unit 27 includes a positioning sensor capable of detecting the current location of the customer service robot 10 within the store. The positioning sensor can be implemented using positioning technology, for example, a GPS sensor. Alternatively, the positioning sensor may be implemented using a position measurement device that utilizes positioning technology such as beacon positioning. In this case, by providing terminals compatible with positioning technology such as beacon positioning at various locations within the store, the current location of the customer service robot 10 can be detected (identified) by the positioning system formed by the terminals and the positioning sensor.

[0028] Furthermore, the sensor unit 27 may include a distance measuring sensor. Multiple distance measuring sensors are provided on the front and back sides of the customer service robot 10, etc., to detect objects around the customer service robot 10 and the distance to those objects. The distance measuring sensor can be implemented using, for example, a sensor device that detects objects and measures distances using captured images or ultrasound, or a sensor device such as LiDAR (Light Detection and ranging) that detects objects using laser light.

[0029] Furthermore, the sensor unit 27 may include a vibration detection sensor capable of detecting contact or impact to the customer service robot 10, a speed sensor for detecting the movement speed of the customer service robot 10, a rudder angle sensor or gyro sensor for detecting the direction of movement, etc. The sensor unit 27 may also include a weight sensor for detecting the weight of products placed on the serving table.

[0030] Figure 3 shows an example of the hardware configuration of a server device 30 according to this embodiment. The server device 30 includes a CPU 31, ROM 32, RAM 33, communication unit 34, and storage unit 35.

[0031] The CPU 31 is an example of a processor that comprehensively controls the operation of the server device 30. The ROM 32 stores various programs. The RAM 33 is a workspace for displaying programs and various data.

[0032] The CPU 31, ROM 32, and RAM 33 are connected via a bus or the like to form a control unit 300 of the computer configuration. In the control unit 300, the CPU 31 operates according to a program stored in the ROM 32 or memory unit 35 and loaded into the RAM 33, thereby executing various processes.

[0033] The communication unit 34 is a wired or wireless communication interface that can be connected to the network N. The communication unit 34 communicates with external devices such as the customer service robot 10 via the network N.

[0034] The storage unit 35 has a storage medium such as an HDD or flash memory, and retains its contents even when the power is cut off. The storage unit 35 stores programs 351 that the CPU 31 can execute, as well as various setting information. For example, the program 351 includes a program for realizing the functional configuration described later.

[0035] Furthermore, the memory unit 35 stores a generation AI 352 for realizing the generation AI function. The generation AI 352 is an example of a generation AI that generates text data, and is implemented, for example, by a large language model (LLM). When a prompt containing instructions is input to the generation AI 352, it generates and outputs text (hereinafter also referred to as a response) corresponding to the instructions in the prompt. For example, the prompt may include instructions on the type of response to be generated, as well as conditions (hereinafter also referred to as constraints) to narrow down the direction and scope of the response.

[0036] For example, when the generating AI 352 receives a prompt instructing it to create a customer service message with constraints such as the state of the customer service robot 10 and the state of customers present around the customer service robot 10, it generates a customer service message that takes these constraints into account as a response.

[0037] The generated AI 352 shall be generated using known deep learning techniques, but the method of generation is not particularly limited. For example, the generated AI 352 may be fine-tuned based on the operating environment of the customer service robot 10 and the characteristics of the store where the customer service robot 10 is used.

[0038] Figure 4 shows an example of the functional configuration of the customer service robot 10 and server device 30 according to this embodiment. For the sake of clarity, Figure 4 also includes some of the hardware configurations described above.

[0039] The control unit 100 of the customer service robot 10 functions as an image acquisition unit 101, a voice recognition unit 102, a sensing result acquisition unit 103, an motion control unit 104, a display control unit 105, a voice output control unit 106, and a customer service support unit 107, with the CPU 11 operating according to a program 151 stored in the ROM 12 or memory unit 15. Some or all of these functions may be implemented by hardware such as dedicated circuits.

[0040] Furthermore, the control unit 300 of the server device 30 functions as a reception unit 301, a prompt generation unit 302, and a response provision unit 303, with the CPU 31 operating according to the program 351 stored in the ROM 32 and the storage unit 35. Some or all of these functions may be implemented by hardware such as dedicated circuits.

[0041] The image acquisition unit 101 of the customer service robot 10 is an example of a first acquisition means. The image acquisition unit 101 sequentially acquires the captured images taken by the imaging unit 22. The image acquisition unit 101 also stores the captured images for a predetermined period of time in RAM 13 or the like.

[0042] The voice recognition unit 102 of the customer service robot 10 recognizes the voice input via the voice input unit 25 and converts the recognized voice into text (text data). It is possible to use publicly known technologies for the voice recognition.

[0043] The sensing result acquisition unit 103 of the customer service robot 10 is an example of a second acquisition means. The sensing result acquisition unit 103 acquires various sensing results sensed by the sensor unit 27.

[0044] The motion control unit 104 of the customer service robot 10 comprehensively controls the movements of the customer service robot 10. Specifically, the motion control unit 104 controls the movement unit 21 (drive unit) and makes the customer service robot 10 move based on the captured images acquired by the image acquisition unit 101 and the various sensing results acquired by the sensing result acquisition unit 103, thereby enabling the customer service robot 10 to move autonomously.

[0045] Furthermore, the motion control unit 104 switches the operating mode of the customer service robot 10 according to instructions from the operation unit 24 or external devices, and the usage scenario of the customer service robot 10. For example, the motion control unit 104 switches the operating mode of the customer service robot 10 to call mode, seating guidance mode, serving mode, etc.

[0046] The customer attraction mode is an operating mode used to attract customers at the entrance of a store, etc. In customer attraction mode, the operation control unit 104 moves the customer service robot 10 to the vicinity of the store entrance and, in cooperation with the customer service support unit 107 (described later), outputs various information related to customer service support.

[0047] The seat guidance mode is an operation mode in which customers who enter the store are guided to an available seat. In seat guidance mode, the operation control unit 104 works in cooperation with the store server, etc., which manages the availability of seats in the store, to move the customer service robot 10 to an available seat while leading the customer. In addition, the operation control unit 104 works in cooperation with the customer service support unit 107, which will be described later, to output various information related to customer service support.

[0048] The serving mode is an operation mode in which the products (food and beverages) ordered by the customer are delivered (served). In serving mode, the operation control unit 104 works in cooperation with the store server, etc., which manages the order status and serving status of each seat, to deliver the products ordered by the customer to the seat where the customer is seated. In addition, the operation control unit 104 works in cooperation with the customer service support unit 107, which will be described later, to output various information related to customer service support.

[0049] The operating modes of the customer service robot 10 are not limited to the examples above. For example, the customer service robot 10 may have an operating mode that guides customers who have finished eating and drinking to the payment area.

[0050] The display control unit 105 of the customer service robot 10 displays various information on the display unit 23 under the control of the motion control unit 104. The voice output control unit 106 of the customer service robot 10 outputs various voices from the voice output unit 26 under the control of the motion control unit 104.

[0051] The customer service support unit 107 of the customer service robot 10 is an example of an estimation means, a third acquisition means, and an output means. The customer service support unit 107 performs various processes related to customer service support using the customer service robot 10. Specifically, the customer service support unit 107 performs processes related to outputting customer service messages according to the state of the customer service robot 10 and the state of the area around the customer service robot 10.

[0052] The customer service support unit 107 estimates (also called discriminating or acquiring) the state of the customer service robot 10 based on information obtained from the imaging unit 22, the sensor unit 27, etc., and acquires the estimation result as state information.

[0053] For example, the customer service support unit 107 determines the location and status of the customer service robot 10, such as whether it is moving or not, based on the sensing results of the positioning sensor and speed sensor acquired by the sensing result acquisition unit 103. For example, the customer service support unit 107 determines the status of the customer service robot 10, such as whether it is loading goods or not, based on the sensing results of the weight sensor acquired by the sensing result acquisition unit 103.

[0054] Furthermore, the customer service support unit 107 infers the state of the customer service robot 10 by combining different types of sensing results and the inference results derived from the captured images, as described below. For example, if the customer service robot 10 is stopped near the entrance of the store, the customer service support unit 107 infers that the customer service robot 10 is in a state of attracting customers (or is in a state where it can attract customers). For example, if the customer service robot 10 is inside the store and is loaded with goods, the customer service support unit 10 infers that the customer service robot 10 is in a state of delivering goods (or is in a state where it can transport goods). For example, if the customer service robot 10 is moving around the store without any goods loaded and the same person is following behind the customer service robot 10, the customer service support unit 10 infers that the customer service robot 10 is guiding customers to empty seats (or is in a state where it can guide customers to empty seats).

[0055] Furthermore, the customer service support unit 107 may, in cooperation with the motion control unit 104, determine the operating mode currently being executed by the customer service robot 10 as the state of the customer service robot 10.

[0056] Furthermore, the customer service support unit 107 estimates the state of the area surrounding the customer service robot 10 based on the captured images acquired by the image acquisition unit 101. Specifically, the customer service support unit 107 estimates (also called acquiring or deriving) the state of people present around the customer service robot 10 based on the captured images, and acquires the estimation result as person information.

[0057] More specifically, the customer service support unit 107 detects areas where people exist (hereinafter referred to as "people areas") from the captured image acquired by the image acquisition unit 101. The customer service support unit 107 acquires the number of detected people as the number of people, and also acquires information indicating the outward appearance of each person from each people area as people information representing the state of people present around the customer service robot 10. For example, the customer service support unit 107 acquires age group, gender, clothing characteristics, etc., from the face, body shape, clothing, etc., shown in the people areas. In addition, based on the location of customer areas present in the captured image, the customer service support unit 107 acquires information that can identify the location of people relative to the customer service robot 10 as people information representing the state of people present around the customer service robot 10.

[0058] Furthermore, there are no particular restrictions on the technology used for detecting the human region, and any known technology can be used. For example, the customer service support unit 107 may detect the human region from the captured image using image recognition technology such as face recognition or skeletal detection. Also, there are no particular restrictions on the method for acquiring human features, and any known technology such as image feature extraction or AI models for estimating age and gender can be used.

[0059] Furthermore, the customer service support unit 107 may acquire not only external features that can be directly read from the captured image, but also the inner state of the person. For example, the customer service support unit 107 may infer the psychological state of the person, such as their emotions, from the facial expressions and clothing features obtained from the captured image, and acquire the inferred psychological state as person information. As an example, if the clothing features shown in the captured image are "short sleeves with rolled-up sleeves," and the facial image in the captured image indicates that the person is sweating, the customer service support unit 107 may infer that the person's psychological state is "hot (or looks hot)."

[0060] Furthermore, there are no particular restrictions on the technology used for inferring psychological states, and publicly known technologies can be used. For example, the customer service support unit 107 may detect the psychological state from captured images of a person by using an AI model trained for emotion recognition or emotion inference. Alternatively, the customer service support unit 107 may infer the psychological state by combining the sensing results of the sensor unit 27.

[0061] When the customer service support unit 107 acquires status information and person information, it generates condition information including the status information and person information and transmits it to the server device 30. Specifically, the customer service support unit 107 transmits the condition information to the server device 30 via the communication unit 14.

[0062] Furthermore, the customer service support unit 107 may include identification information (hereinafter also referred to as a robot ID) that can identify its own customer service robot 10 in the condition information. In addition, the customer service support unit 107 may include information in the condition information regarding the type of store where its customer service robot 10 is located (for example, a restaurant), the products (food and beverages) that can be offered at the store, and the product being served.

[0063] Furthermore, there are no specific restrictions on the timing of sending condition information, and it is possible to set it arbitrarily. For example, the customer service support unit 107 may send condition information when it acquires person information. Alternatively, the customer service support unit 107 may suppress the retransmission of condition information until the status information changes while the same person is detected.

[0064] Meanwhile, the reception unit 301 of the server device 30 receives condition information transmitted from the customer service robot 10 via the communication unit 34.

[0065] The prompt generation unit 302 of the server device 30 generates a prompt that serves as instruction information for the generation AI 352 based on the condition information received by the reception unit 301. The prompt generation unit 302 then inputs the generated prompt to the generation AI 352.

[0066] For example, the prompt generation unit 302 generates prompts corresponding to the state of the customer service robot 10 based on the state information included in the condition information. As an example, if the customer service robot 10 is stopped at the entrance of the store, or is operating (or capable of operating) in calling mode, the prompt generation unit 302 generates a prompt instructing the creation of a calling message. Also, if the customer service robot 10 is moving towards an empty seat in the store, or is operating (or capable of operating) in seat guidance mode, the prompt generation unit 302 generates a prompt instructing the creation of a seat guidance message. Furthermore, if the customer service robot 10 is loading food and serving, or is operating (or capable of operating) in serving mode, the prompt generation unit 302 generates a prompt instructing the creation of a serving message.

[0067] Furthermore, the prompt generation unit 302 generates prompts corresponding to the state of the customer service robot 10 and the surrounding conditions, based on the state information and person information included in the condition information. For example, if the customer service robot 10 is moving around the store and there is a person in the direction of the customer service robot 10's movement, the prompt generates a prompt instructing the robot to create a message asking the person to move aside.

[0068] Furthermore, the prompt generation unit 302 generates prompts that use the person information included in the condition information as constraints when generating the response text. For example, the prompt generation unit 302 generates prompts that use the number of people, age group, gender, number of people, emotions, etc. included in the person information as constraints. Also, if the condition information includes information such as the type of business of the store, the products that can be offered at the store, or the products being served, the prompt generation unit 302 generates prompts that use that information as constraints.

[0069] The method for generating prompts is not particularly limited. For example, template prompts for each state of the customer service robot 10 may be stored in advance in the memory unit 35, and the customer service support unit 107 may use different template prompts depending on the state information included in the condition information. Alternatively, the customer service support unit 107 may generate prompts by combining multiple constraint conditions.

[0070] When the generation AI 352 receives a prompt from the prompt generation unit 302, it generates a response sentence corresponding to the instructions of that prompt.

[0071] For example, when the generating AI352 receives a prompt containing the following content (A) as a prompt to instruct it to create a call message, it generates the response sentence (B). (A) This is a restaurant. Please create a message of 300 characters or less to attract customers, under the following constraints. ・A total of 5 people: 3 men and 2 women Looks hot. • Recommend beer (B) Welcome! Thank you for coming. There are 5 of you in your party: 3 men and 2 women. It's a hot day today, isn't it? If you want to feel refreshed, I recommend a cold beer. We have seats available where you can relax in a cool environment. Please feel free to ask us for anything.

[0072] The constraint that recommends beer included in prompt (A) may be specified in the condition information, or it may be added by the prompt generation unit 302 from among the products available at the store based on the constraint that it is "hot".

[0073] Furthermore, for example, when the generating AI352 receives a prompt containing the content of (C) below, which instructs it to create a message to ask someone to move aside, it generates the response sentence (D). (C) I want to move forward, but there is a customer in my way. Please create a message of 100 characters or less to ask the customer to move aside, given the following constraints. • Telling young children (D) Excuse me. Please let me move forward. Thank you!

[0074] Furthermore, for example, when the generating AI352 receives a prompt with the content of (E) below, which is a prompt of the same meaning but with different constraints than (C) above, it generates the response sentence (F). (E) You want to move forward, but a customer is in your way. Please create a message of 100 characters or less to ask the customer to move aside, given the following constraints. • Telling adults (F) Excuse me. May I move forward? I apologize for the inconvenience, but thank you.

[0075] The response provision unit 303 of the server device 30 transmits the response text generated by the generation AI to the customer service robot 10 via the communication unit 34. Specifically, the response provision unit 303 transmits the response text generated based on the condition information to the customer service robot 10 that sent the condition information.

[0076] The customer service support unit 107 of the customer service robot 10 receives a response message transmitted from the server device 30 via the communication unit 34, and then works in cooperation with the motion control unit 104 to output information based on the response message. Specifically, the customer service support unit 107 displays a message or screen corresponding to the response message on the display unit 23. In addition, the customer service support unit 107 outputs audio corresponding to the response message from the audio output unit 26.

[0077] In this way, the customer service robot 10 can present a response message, which is a customer service message generated according to the state of the customer service robot 10 and the surrounding conditions, to a person such as a customer, thus enabling efficient customer service support.

[0078] Furthermore, the customer service support unit 107 may change the method of outputting the response text depending on the status (characteristics) of the person acquired as person information.

[0079] For example, if the person is a toddler, the customer service support unit 107 will output the response using a child-friendly voice and hiragana. If the person is an adult, the customer service support unit 107 will output the response using a calm tone and kanji. If the person is an elderly person, the customer service support unit 107 will output the response by increasing the volume of the voice or repeating the voice output twice. By changing the method of outputting the response according to the person's state in this way, the level of understanding of the presented content can be improved, making customer service support more efficient.

[0080] The following describes an example of the operation of the customer service support system 1. Figure 5 is a sequence diagram showing an example of the processing performed by the customer service robot 10 and the server device 30 of the embodiment. It should be noted that, as a prerequisite for this processing, the customer service robot 10 is operating under the control of the motion control unit 104.

[0081] First, the customer service support unit 107 of the customer service robot 10 acquires condition information representing the state of the customer service robot 10 and the surrounding environment based on information such as images captured by the imaging unit 22 and sensing results from the sensor unit 27 (step S11). Next, the customer service support unit 107 transmits the acquired condition information to the server device 30 (step S12).

[0082] In the server device 30, when the reception unit 301 receives condition information transmitted from the customer service robot 10, the prompt generation unit 302 generates a prompt corresponding to the condition information (step S21). Next, the prompt generation unit 302 inputs the generated prompt to the generation AI 352, causing the generation AI 352 to generate a response (step S22). Then, the response provision unit 303 provides the generated response to the customer service robot 10, which was the source of the condition information (step S23).

[0083] When the customer service support unit 107 of the customer service robot 10 obtains a response from the server device 30, it outputs information based on that response via the display unit 23 and the voice output unit 26 (step S13).

[0084] Through the above process, the customer service robot 10 and the server device 30 can output response messages that correspond to the state of the customer service robot 10 and the surrounding conditions, in other words, customer service messages that correspond to the usage scenario of the customer service robot 10 and the state of people in the surrounding area. As a result, the customer service robot 10 can output messages appropriate to the usage scenario to customers in its vicinity, thereby enabling efficient customer service support.

[0085] Next, with reference to Figure 6, an example of the processing performed by the customer service robot 10 will be described. Here, Figure 6 is a flowchart of an example of the processing performed by the customer service robot 10 of the embodiment. Note that this processing is a more detailed description of the processing in steps S11 to S13 described in Figure 5.

[0086] The customer service support unit 107 waits until it detects a person in the vicinity of the customer service robot 10, based on information such as the image captured by the imaging unit 22 and the sensing results from the sensor unit 27 (Step S31; No). For example, the customer service support unit 107 waits until it detects a person's area from the captured image.

[0087] When the customer service support unit 107 detects a person in the vicinity of the customer service robot 10 (Step S31; Yes), it acquires person information indicating the state of that person from the captured image (Step S32), and also acquires state information indicating the state of the customer service robot 10 from the sensing results, etc. (Step S33). Next, the customer service support unit 107 generates condition information including the state information and person information, and transmits the generated condition information to the server device 30 (Step S34).

[0088] Next, the customer service support unit 107 obtains the response text from the server device 30 (step S35), and then determines how to output the response text based on the person information obtained in step S32 (step S36). For example, the customer service support unit 107 determines the tone of voice and repetition count when outputting the response text as audio, and the display method when outputting the response text on paper, according to the age group of the person. Then, the customer service support unit 107 outputs the response text using the output method determined in step S37 (step S37).

[0089] As described above, the customer service robot 10 acquires captured images from the imaging unit 22 that images the area around the customer service robot 10, and also acquires sensing results from the sensor unit 27 that senses information related to the operation of the customer service robot 10. Furthermore, the customer service robot 10 estimates the state of the customer service robot 10 and the state of the area around the customer service robot 10 based on the captured images and sensing results, and transmits (inputs) the estimation results as condition information to the server device 30, thereby acquiring the response text generated by the server device 30. The customer service robot 10 then outputs the acquired response text via the display unit 23 and the voice output unit 26.

[0090] As a result, the customer service robot 10 can present customers or other individuals with response messages (customer service messages) that are appropriate to the usage scenario of the customer service robot 10, such as attracting customers or guiding them to their seats, thereby enabling efficient customer service support. Furthermore, the customer service robot 10 can attract the attention and interest of people in its vicinity by presenting response messages that are appropriate to their state, thereby enabling efficient customer service support.

[0091] Furthermore, the customer service robot 10 estimates its own state and the surrounding conditions based on various information acquired by the imaging unit 22 and the sensor unit 27, and transmits conditional information including the estimation results to the server device 30. As a result, in this embodiment, compared to a configuration in which the captured images acquired by the customer service robot 10 are transmitted to the server device 30 and the server device 30 is made to estimate the state, the amount of data involved in communication and the processing load on the server device 30 can be reduced. Therefore, in the customer service support system 1 of this embodiment, the real-time nature of the output of response sentences can be improved, and customer service support can be performed efficiently.

[0092] The above-described embodiments can also be modified and implemented as appropriate by changing some of the configurations or functions of each of the above-described devices. Therefore, several modifications of the above-described embodiments will be described below as other embodiments. In the following, we will mainly describe the differences from the above-described embodiments, and will omit detailed explanations of points that are common with what has already been described. Furthermore, the modifications described below may be implemented individually or in combination as appropriate.

[0093] (Variation 1) In the above-described embodiment, a method for acquiring the state of the customer service robot 10 and the state of the area around the customer service robot 10 was explained, but the invention is not limited to this, and for example, the state of the area around the store where the customer service robot 10 is placed may also be acquired.

[0094] For example, the customer service support unit 107 of the customer service robot 10 may function as an example of a fourth acquisition means to acquire environmental information such as the weather and temperature around the store where its customer service robot 10 is located from an external device (for example, an external server that provides weather information) via the communication unit 14. In this case, the customer service support unit 107 may use the environmental information together with the aforementioned person information, etc., to infer the customer's emotions. The customer service support unit 107 may also include the environmental information in the condition information and transmit it to the server device 30.

[0095] Furthermore, when environmental information is included in the condition information and transmitted to the server device 30, it is preferable that the prompt generation unit 302 of the server device 30 generates a prompt that takes the environmental information into account. Specifically, the prompt generation unit 302 generates a prompt that includes the weather and temperature included in the environmental information as constraint conditions.

[0096] For example, if the prompt generation unit 302 includes environmental information indicating that the weather is "rainy" in the condition information related to the prompt in (A) described above, it generates the prompt in (G) below. (G) This is a restaurant. Please create a message of 300 characters or less to attract customers, under the following constraints. ・A total of 5 people: 3 men and 2 women Looks hot. • Recommend beer ·rain

[0097] In this case, the generating AI352 generates the response sentence (H) in response to the input prompt for the content of (G). (H) Welcome! Thank you for coming despite the bad weather. There are 5 of you in your party: 3 men and 2 women. It's quite humid today, isn't it? If you want to feel refreshed, I recommend a cold beer. We have seats available where you can relax in the cool air. Please feel free to ask us for anything.

[0098] As a result, the customer service robot 10 achieves the same effects as the embodiment described above, and can output response messages that take into account the state of the environment experienced by the customer, thereby presenting more persuasive customer service messages to the customer. Therefore, the customer service robot 10 can provide customer service support more efficiently.

[0099] (Modification 2) In the above-described embodiment, a method for acquiring the state of the customer service robot 10 and the state of the area around the customer service robot 10 was explained, but the invention is not limited to this, and for example, the state of the store where the customer service robot 10 is located may also be acquired.

[0100] For example, the customer service support unit 107 of the customer service robot 10 may function as an example of a fourth acquisition means to acquire the availability status of seats in the store where its customer service robot 10 is located from an external device (such as a store server that manages seat availability) via the communication unit 14. In this case, the customer service support unit 107 may only transmit condition information to the server device 30 if it determines that it is in a state to attract customers and there are seats available in the store. Alternatively, the customer service support unit 107 may include the availability status of seats in the store in the condition information and transmit it to the server device 30.

[0101] Furthermore, when sending the seat availability status as condition information to the server device 30, it is preferable that the prompt generation unit 302 of the server device 30 generates a prompt that takes the seat availability status into account. For example, the prompt generation unit 302 generates a prompt that uses the presence or absence of seats as a constraint condition. In this case, for example, when a prompt is input that adds the constraint condition of "no seats available" to the prompt described in (A) above, the generation AI 352 generates a response statement that indicates that the place is currently full and that they can contact you as soon as a seat becomes available.

[0102] As a result, the customer service robot 10 achieves the same effects as the embodiment described above, and can output response messages that take into account the store's condition, thus enabling it to present more practical customer service messages to customers. Therefore, the customer service robot 10 can provide customer service support more efficiently.

[0103] (Variation 3) In the above-described embodiment, a form was explained in which state information and person information obtained from captured images and sensing results are included in the condition information. However, the invention is not limited to this, and the content of speech spoken to the customer service robot 10 may also be included in the condition information.

[0104] Specifically, the customer service support unit 107 extracts speech content spoken to the customer service robot 10 from the text recognized by the speech recognition unit 102 as speech information. For example, the customer service support unit 107 determines whether the text contains speech content spoken to the customer service robot 10 based on the orientation of the person's face and mouth movements in the captured image, the content of the text recognized by the speech recognition unit 102, etc. If the customer service support unit 107 determines that the text contains speech content spoken to the customer service robot 10, it extracts the speech content as speech information. The customer service support unit 107 then transmits the extracted speech information, along with state information and person information, to the server device 30 as condition information.

[0105] Furthermore, there are no particular restrictions on the technology used for extracting speech content, and any known technology can be used. For example, the customer service support unit 107 may use a trained AI model for conversation analysis to extract the speech content spoken to the customer service robot 10.

[0106] Meanwhile, the prompt generation unit 302 of the server device 30 generates prompts that include the utterance content contained in the utterance information, based on the utterance information contained in the condition information. For example, the prompt generation unit 302 generates prompts that instruct the generation of a response sentence regarding the utterance content, or generates prompts that use the items contained in the utterance content as constraints.

[0107] As a result, the customer service robot 10 can achieve the same effects as the embodiment described above, and can also provide responses in response to inquiries from customers, thus enabling more efficient customer service support.

[0108] (Modification 4) In the above embodiment, the server device 30 generates the prompt, but the system is not limited to this, and the customer service robot 10 may generate the prompt. In this case, for example, the customer service robot 10 is equipped with a prompt generation unit 302, which generates a prompt based on the condition information generated by the customer service support unit 107. The prompt generation unit 302 of the customer service robot 10 also transmits the generated prompt to the server device 30, which then inputs the prompt to the generation AI 352 via the reception unit 301.

[0109] This allows for the same effects as the above-described embodiment, and also enables the customer service robot 10 to be equipped with a prompt generation function, thus allowing for a flexible system configuration of the customer service support system 1.

[0110] (Variation 5) In the embodiments described above, customer service at a store selling goods (food and beverages) was used as an example, but the system is not limited to this and can be applied to customer service in other industries. For example, the customer service system described above can be applied to industries such as hotels that provide accommodation services.

[0111] (Experimental variation 6) In the above-described embodiment, a customer service robot 10 was used as an example of a customer service support device. However, the invention is not limited to this device, and can be applied to other devices as long as they serve as a point of contact (touch point) with customers.

[0112] For example, another example of a customer service support device is its application to devices such as automatic doors installed at the entrances and exits of stores. In this case, the imaging unit 22 of the automatic door is installed, for example, on the outside and inside of the store to image customers entering and leaving the store. The sensor unit 27 of the automatic door includes a motion sensor or the like.

[0113] The automatic door's operation control unit 104 controls the opening and closing of the door according to the sensing results of the human presence sensor. The operation control unit 104 also outputs customer service voice messages via the voice output unit 26 according to the status of its own automatic door. When the customer service support device is an automatic door, the display unit 23 and the operation unit 24 may be removed or attached externally in the form of a signage terminal or the like.

[0114] Furthermore, the customer service support unit 107 of the automatic door determines (predicts) the state of the automatic door based on information obtained from the imaging unit 22, the sensor unit 27, etc., and generates state information indicating the determination result.

[0115] For example, the customer service support unit 107 determines that it is time to call in customers if a person is present in the image captured outside the store and the motion sensor does not detect that person. For example, the customer service support unit 107 determines that it is time to guide customers entering the store if a person is present in the image captured outside the store and the motion sensor detects that person. For example, the customer service support unit 107 determines that it is time to guide customers leaving the store if a person is present in the image captured inside the store and the motion sensor detects that person.

[0116] The customer service support unit 107 will acquire person information in the same manner as in the embodiment described above, but the opportunities for acquiring person information may be limited, for example, when calling out to customers. Also, the server device 30 will generate a prompt based on the condition information transmitted from the automatic door (customer service support device), and will transmit the response text obtained by inputting the prompt into the generation AI 352 to the automatic door, in the same manner as in the embodiment described above.

[0117] As a result, the automatic door can output a response message, i.e., a customer service message, in voice or other format, corresponding to the status of the automatic door itself, thus achieving the same effect as the embodiment described above. It is possible.

[0118] While embodiments (and modifications) of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, combinations, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0119] Furthermore, the programs executed by each device in the above embodiments are provided pre-installed in the storage medium (ROM or storage unit) provided by the device, but are not limited to this. For example, the programs may be configured to be provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk). Moreover, the storage medium is not limited to a medium independent of the computer or embedded system, but also includes storage media that have downloaded and stored or temporarily stored programs transmitted via a LAN, the Internet, etc.

[0120] Furthermore, the programs executed by each device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by allowing users to download them via the network, or they may be configured to be provided or distributed via a network such as the Internet. [Explanation of Symbols]

[0121] 1. Customer service support system 10 Customer service support equipment 21 Mobile unit 22 Imaging Department 23 Display section 24 Control section 25 Voice input section 26 Audio output section 27. Recovery Department 30 Server Devices 100 Control Unit 101 Image acquisition unit 102 Voice Recognition Unit 103 Sensing Result Acquisition Unit 104 Operation Control Unit 105 Display Control Unit 106 Audio Output Control Unit 107 Customer Service Support Department 300 Control Unit Room 301 Reception 302 Prompt generation unit 303 Answer provision department 352 Generation AI [Prior art documents] [Patent Documents]

[0122] [Patent Document 1] Japanese Patent Publication No. 2017-117184

Claims

1. It is a customer service support device that serves as a point of contact with customers, A first acquisition means for acquiring captured images from an imaging device that captures images around the customer service support device, A second acquisition means for acquiring sensing results from a sensor device that senses information related to the operation of the customer service support device, An estimation means for inferring the state of the customer service support device and the state of the surrounding area of ​​the customer service support device based on the captured image and the sensing results, A third acquisition means obtains the response text generated by a generation device, which has a generation AI function that generates customer service response text according to the conditions of the input state, by inputting the prediction result of the prediction means. Output means for outputting the aforementioned response text, A customer service support device equipped with the following features.

2. The estimation means estimates the state of people present around the customer service support device based on the captured image. The customer service support device according to claim 1.

3. The estimation means estimates the state of the operation performed by the customer service support device based on the captured image and the sensing results. The customer service support device according to claim 1 or 2.

4. The output means changes the method of outputting the response text according to the state of the person inferred by the inference means. The customer service support device according to claim 2.

5. The system further comprises a fourth acquisition means for acquiring information regarding the state of the store where the customer service support device is installed or the state of the surrounding environment of the store. The third acquisition means inputs the information acquired by the fourth acquisition means together with the estimation result to the generation device, thereby acquiring the answer sentence generated by the generation device. The customer service support device according to claim 1.

6. The computer for the customer service support device, which serves as a point of contact with customers, A first acquisition means for acquiring captured images from an imaging device that captures images around the customer service support device, A second acquisition means for acquiring sensing results from a sensor device that senses information related to the operation of the customer service support device, An estimation means for inferring the state of the customer service support device and the state of the surrounding area of ​​the customer service support device based on the captured image and the sensing results, A third acquisition means obtains the response text generated by a generation device, which has a generation AI function that generates customer service response text according to the conditions of the input state, by inputting the prediction result of the prediction means. Output means for outputting the aforementioned response text, A program to make it work.

Images