Information processing device and information processing method

Abstract

To increase the convenience of promotion to a customer who uses a generative AI.SOLUTION: An information processing device includes an acquisition unit, a detection unit, a first generation unit, a second generation unit, and an output control unit. The acquisition unit obtains, from a camera installed in a facility, a camera image captured by the camera. The detection unit detects a feature of a customer in the camera image. The first generation unit generates first text information indicating the feature of the customer based on the detected feature. The second generation unit inputs inquiry text with the first text information added into a generative AI for generating text relating to merchandise which is handled in the facility according to an input of inquiry text; and generates second text information relating to promotion of the merchandise based on text generated by the generative AI. The output control unit outputs the second text information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an information processing device and an information processing method. [Background technology]

[0002] In recent years, generative AI (Artificial Intelligence) that generates text, images, etc. has been attracting attention. The accuracy of generative AI is improving every day, and generative AI that generates text is now being used to promote products that match customer inquiries.

[0003] However, promotional services using the above-mentioned generation AI are generally provided via the cloud. Therefore, if a service handles information that should not be output to the cloud, such as customer personal information, it may not be possible to use the service. Furthermore, since the generation AI used in promotional services provided via the cloud is not optimized for stores, there is a possibility that the service may recommend products that are not available in the store. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an information processing device and an information processing method that can improve the convenience of promotions for customers using generation AI. [Means for solving the problem]

[0005] The information processing device according to the embodiment includes: [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a system diagram showing an example of the connection relationships of devices in an information processing system according to an embodiment. [Figure 2]FIG. 2 is a block diagram showing an example of the hardware configuration of the SC according to the embodiment. [Figure 3] FIG. 3 is a block diagram showing an example of the functional configuration of the SC according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of processing executed by the SC according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] An information processing device and an information processing method according to an embodiment will be described in detail below with reference to Figures 1 to 4. In the embodiment described below, a store computer (SC) installed in a store (an example of a facility) such as a department store or supermarket will be described as an example of an information processing device, but the present invention is not limited to the embodiment.

[0008] 1 is a system diagram showing an example of the connection relationships of devices in an information processing system S according to an embodiment. In FIG. 1, the system includes an SC (Store Computer) 1, a plurality of image display devices 2, a plurality of audio output devices 3, a plurality of cameras 4, and a plurality of mobile terminals 6.

[0009] SC1, a plurality of image display devices 2, a plurality of audio output devices 3, and a plurality of cameras 4 are connected to one another via a communication line 7 such as a LAN (Local Area Network). In addition, a plurality of mobile terminals 6 are connected to SC1 via an access point 5, which is a repeater for wireless communication, and the communication line 7.

[0010] 1 is an example, and the number of devices included in the information processing system S is not limited to the number shown in Fig. 1. For example, the information processing system S may include a plurality of access points 5.

[0011] The image display device 2 displays a promotional image for promoting a product that is recommended for purchase to a customer. For example, the image display device 2 is an information processing device having a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. Note that the image display device 2 may be the display device itself.

[0012] For example, the image display device 2 receives a promotional image generated by the SC1 from the SC1 via the communication line 7, and controls the display device to display the promotional image.

[0013] The audio output device 3 outputs promotional audio for promoting products recommended for purchase to customers. For example, the audio output device 3 is an information processing device having an audio output device such as a speaker. Note that the audio output device 3 may be the audio output device itself.

[0014] For example, the audio output device 3 receives promotional audio generated by the SC1 from the SC1 via the communication line 7, and controls the audio output device to output the promotional audio.

[0015] SC1 generates promotional information (promotional images, promotional audio) for customers. For example, SC1 is a server device. SC1 may also perform processes such as collecting and managing product sales registration information received from a POS terminal (not shown). SC1 may be configured as a single server device or multiple server devices.

[0016] For example, SC1 receives camera images of the inside of the store captured by camera 4 from camera 4 via communication line 7. Also, for example, SC1 receives from mobile terminal 6 the reception results of radio waves constantly transmitted from multiple transmitters (for example, radio wave transmitters such as Bluetooth (registered trademark) and Wi-Fi) (not shown) installed in the store.

[0017] Also, for example, the SC1 transmits the generated promotional image to the image display device 2. Also, for example, the SC1 transmits the generated promotional sound to the sound output device 3.

[0018] Camera 4 captures images including people inside the store. For example, multiple cameras 4 are installed at regular intervals along the aisles, such as on the ceiling near the aisles, so that they can capture images of people passing through the store.

[0019] As an example, multiple installed cameras 4 capture images of the path a person takes from the time they enter the store until they leave. In the example of Fig. 1, n cameras 4 are installed along the aisles in the store, and these cameras 4 capture images of person PA and person PB. The images captured by the cameras 4 include information indicating the time of capture and the capture position (for example, the installation position of the camera 4 that captured the image).

[0020] The mobile terminal 6 is a device carried by a customer and exchanges various information with the SC1. The mobile terminal 6 is, for example, a smartphone or a tablet terminal. For example, the mobile terminal 6 receives radio waves from multiple transmitters installed in the store. The mobile terminal 6 transmits the radio wave reception results to the SC1 via the access point 5 and the communication line 7.

[0021] Next, the hardware configuration of the SC 1 will be described with reference to Fig. 2, which is a block diagram showing an example of the hardware configuration of the SC 1.

[0022] As shown in Figure 2, SC1 includes a CPU (Central Processing Unit) 11 that serves as the control body, a ROM (Read Only Memory) 12 that stores various programs, a RAM (Random Access Memory) 13 that expands various data, and a memory unit 14 that stores various programs.

[0023] The CPU 11, ROM 12, RAM 13, and memory unit 14 are connected to one another via a data bus 15. The CPU 11, ROM 12, and RAM 13 constitute a control unit 100. That is, the control unit 100 executes various processes by the CPU 11 operating in accordance with a control program 141 stored in the ROM 12 or memory unit 14 and loaded into the RAM 13. The various processes will be described later.

[0024] The RAM 13 develops various programs including the control program 141, and also temporarily stores images captured by the camera 4 until they are stored in the memory unit .

[0025] The memory unit 14 is a non-volatile memory such as a hard disk drive (HDD) or flash memory that retains stored information even when the power is turned off, and stores programs including a control program 141. The memory unit 14 also has a promotion LLM 142, a voice conversion LLM 143, a personal information DB 144, a nearby product DB 145, and a product image DB 146.

[0026] The promotion LLM 142 is a generation AI that generates text and is a large language model (LLM). The promotion LLM 142 generates recommended product text that presents products recommended for purchase to customers. Note that, although an LLM is used as the generation AI in this embodiment, the generation AI is not limited to an LLM as long as it is capable of generating text.

[0027] For example, the recommended product text is text data that describes the name of a product available in a store and the promotion details for that product. Note that the recommended product text may also be text data in a list format that includes the names of multiple products and the promotion details for each of the multiple products.

[0028] For example, the promotion LLM 142 is an LLM constructed using known deep learning technology that, in response to input of text specifying conditions, outputs text related to a product based on the conditions. Here, the conditions may be, for example, reference conditions that serve as references for deriving the output results or constraint conditions for narrowing down the output results.

[0029] In this embodiment, the promotion LLM 142 generates recommended product text for a customer in response to an input of an inquiry text that includes conditions related to the customer's location information, customer characteristic information, and customer personal information, etc. The customer's location information, customer characteristic information, and customer personal information will be described later.

[0030] The speech conversion LLM 143 is a generation AI that generates text, and is an LLM. The speech conversion LLM 143 generates promotional speech text, which is text data that serves as the basis for promotional speech. For example, the promotional speech text is text that encourages customers to purchase a product using expressions that would be considered natural if explained verbally.

[0031] For example, the speech conversion LLM 143 is an LLM constructed using known deep learning technology that outputs text for promoting a product in response to input of text related to the product.

[0032] The speech conversion LLM 143 of this embodiment generates promotional speech text for a customer in response to input of text data in which the customer's location information, customer characteristic information, and personal information about the customer are added to the product text.

[0033] The personal information DB (Data Base) 144 manages the personal information of customers. For example, the personal information includes customer information such as the customer's address, name, age, sex, customer ID, telephone number, email address, facial image, information on registered mobile devices, product purchase history at stores, and output history of recommended product text.

[0034] As an example, personal information DB144 is a database that stores, for each customer, the customer's address, name, age, gender, customer ID, telephone number, email address, facial image, identification information of the registered mobile device (terminal ID), product purchase history at the store, and history of recommended product text, all in association with each other.

[0035] The nearby product DB 145 is a database that defines products located near each location within a store. As an example, the nearby product DB 145 is a database that stores product information in association with each piece of location information within a store. Note that multiple pieces of product information may be associated with one piece of location information. Also, the same product information may be associated with different pieces of location information.

[0036] The product image DB 146 is a database for managing product images. As an example, the product image DB 146 is a database that stores a product ID, a product name, and a product image for each product in association with each other. The product image may include imaged information about the product, such as text introducing the product. The product image may also be a plurality of images of the product taken from various angles.

[0037] An operation unit 17 and a display unit 18 are also connected to the data bus 15 via a controller 16 .

[0038] The operation unit 17 receives various inputs from an operator such as a store clerk, etc. For example, the operation unit 17 includes a numeric keypad for entering numbers, various function keys, and the like.

[0039] Display unit 18 displays various types of information. For example, display unit 18 may display images of the inside of the store input from camera 4. Display unit 18 may also display images captured by a specific camera 4. Display unit 18 may also divide the screen and display images captured by multiple cameras 4 on the same screen at the same time.

[0040] The data bus 15 is also connected to a communication I / F 19 such as a LAN I / F (Interface). The communication I / F 19 is connected to the communication line 7.

[0041] The communication I / F 19 transmits and receives various types of information. For example, the communication I / F 19 receives images captured by the camera 4 in real time.

[0042] Next, the functional configuration of SC1 will be described. Fig. 3 is a functional block diagram showing an example of the functional configuration of SC1. The control unit 100 functions as an LLM pre-processing unit 101, an LLM processing unit 102, and an LLM post-processing unit 103 by following various programs including a control program 141 stored in the ROM 12 and the memory unit 14.

[0043] First, we will explain the LLM preprocessing unit 101. The LLM preprocessing unit 101 performs LLM preprocessing. For example, the LLM preprocessing is a series of processes that, in the process of generating promotion information using the promotion LLM 142, brings the process of inputting query text into the promotion LLM 142 into an executable state.

[0044] As shown in FIG. 3, the LLM preprocessing unit 101 includes an acquisition unit 111, a first identification unit 112, a detection unit 113, a second identification unit 114, and a first generation unit 115.

[0045] The acquisition unit 111 acquires camera images captured by the cameras 4. For example, the acquisition unit 111 acquires the camera images by receiving the camera images from the multiple cameras 4 in real time via the communication line 7. The acquisition unit 111 sends the acquired camera images to the first identification unit 112, the detection unit 113, and the second identification unit 114.

[0046] The first identification unit 112 identifies the personal information of the customer. For example, it extracts a facial image of the customer from the camera image sent from the acquisition unit 111 using a known image recognition technology. The first identification unit 112 refers to the personal information DB 144 stored in the memory unit 14, and identifies the address, name, age, gender, customer ID, telephone number, email address, product purchase history at the store, and output history of recommended product text corresponding to the extracted facial image as the personal information of the customer corresponding to the facial image.

[0047] Furthermore, the first identification unit 112 associates the identified customer ID with the extracted face image and sends them to the detection unit 113. The first identification unit 112 also associates the identified customer ID with the identified information on the registered mobile terminal and sends them to the second identification unit 114.

[0048] Furthermore, first identification unit 112 associates the identified customer ID with the identified personal information and sends them to first generation unit 115 and determination unit 117. Note that the personal information sent to first generation unit 115 and determination unit 117 may be a part of the identified personal information. Furthermore, the type of personal information sent to first generation unit 115 may be different from the type of personal information sent to determination unit 117.

[0049] Here, it can be said that the first identifying unit 112 identifies the customer's product purchase history at the store, and therefore the first identifying unit 112 is an example of a second obtaining unit.

[0050] The detection unit 113 detects characteristic information of a customer captured in a camera image. For example, the characteristic information is information that represents the customer's external characteristics. Specifically, it is information that represents the customer's clothing (for example, wearing a white sweater from brand XX and black pants from brand XX). The characteristic information may also include information about the customer's emotions, such as whether the customer is happy, angry, sad, or happy, as inferred from the customer's facial expression.

[0051] For example, the detection unit 113 identifies a customer corresponding to the facial image sent from the first identification unit 112 from the camera image sent from the acquisition unit 111. The detection unit 113 extracts external features of the identified customer using a known image recognition technique. The detection unit 113 associates the extracted features with the customer ID associated with the facial image sent from the first identification unit 112 as customer feature information and sends it to the first generation unit 115 and the determination unit 117.

[0052] The second identification unit 114 identifies the customer's location information. For example, the location information is information that indicates the customer's location within the store. For example, the second identification unit 114 receives a radio wave reception result from a mobile terminal 6 present within the store via the communication line 7. Here, it is assumed that the radio wave reception result is associated with a terminal ID.

[0053] For example, the second identification unit 114 checks the terminal ID associated with the reception result of the received radio waves, and identifies the radio wave reception result associated with the terminal ID that matches the terminal ID sent from the first identification unit 112. Furthermore, the second identification unit 114 identifies the location of the mobile terminal 6 identified by the terminal ID from information included in the radio wave reception result that indicates from which transmitter in the store the radio waves are being received and at what strength.

[0054] The second identification unit 114 sends information representing the location of the identified mobile terminal 6 as customer location information to the first generation unit 115 and the judgment unit 117, correlating it with the customer ID associated with the terminal ID sent from the first identification unit 112.

[0055] Here, the customer location information sent to the first generation unit 115 and the determination unit 117 includes information indicating the time of reception of the radio waves included in the radio wave reception result by the mobile terminal 6 corresponding to the location information (hereinafter also referred to as the time of the location information). This makes it possible to identify the time at which the customer location information sent to the first generation unit 115 and the determination unit 117 represents the customer location information.

[0056] However, customers do not always enter the store with their mobile terminal 6, and there is also the possibility that for some reason the mobile terminal 6 is unable to receive radio waves from a transmitter installed in the store. Therefore, if the radio wave reception result of the mobile terminal 6 corresponding to the terminal ID sent from the first identification unit 112 cannot be obtained or the position of the mobile terminal 6 cannot be accurately identified, the second identification unit 114 identifies the customer's position information based on the camera image.

[0057] For example, if the radio wave reception result of the mobile terminal 6 corresponding to the terminal ID sent from the first identification unit 112 cannot be obtained, the second identification unit 114 sends the customer ID associated with the terminal ID to the first identification unit 112 and requests that the first identification unit 112 send a facial image corresponding to the customer ID. Upon receiving the request, the first identification unit 112 associates the customer ID with the facial image corresponding to the customer ID and sends them to the second identification unit 114.

[0058] The second identification unit 114 searches the multiple camera images sent from the acquisition unit 111 and identifies a camera image in which the customer corresponding to the facial image sent from the first identification unit 112 was captured. The second identification unit 114 identifies the position of the customer corresponding to the facial image sent from the first identification unit 112 from information indicating the imaging position included in the identified camera image.

[0059] If a customer corresponding to the facial image sent from first identification unit 112 appears in multiple camera images, second identification unit 114 refers to information indicating the image capture time included in the camera images, and identifies the position of the customer corresponding to the facial image sent from first identification unit 112 from information indicating the image capture position included in the latest camera image. In this case, the time of the position information is set to the capture time of the latest camera image.

[0060] The first generation unit 115 generates an inquiry text based on the customer's characteristic information. The inquiry text is instruction information (prompt) that instructs the generation AI on the content of the text to be generated. Hereinafter, the content instructed by the instruction information will also be referred to as the instruction content.

[0061] For example, the first generation unit 115 instructs the promotion LLM 142 to generate text including the name of the product and the content promoting the product, and generates a query text that describes the content indicating that the recommended products are products sold in stores and that the recommended products are derived from conditions based on the processing results of the first identification unit 112, the detection unit 113, and the second identification unit 114.

[0062] Here, templates for generating instruction information are stored in advance in the memory unit 14 or the like. Note that a plurality of templates may be stored in the memory unit 14 or the like. In this case, the first generating unit 115 may use different templates depending on the situation. For example, the first generating unit 115 may switch templates depending on keywords (e.g., words that can identify the product genre (food, clothing, etc.)) included in the query text.

[0063] In addition, the first generation unit 115 adds text data based on the processing of the first identification unit 112, the detection unit 113, and the second identification unit 114 to the query text according to the conditions specified in the instruction content.

[0064] For example, the first generating unit 115 adds, as a condition to the query text, text data including information associated with a customer ID and sent from the first identifying unit 112, the detecting unit 113, and the second identifying unit 114. Here, since each piece of information is associated with a customer ID, it is possible to prevent information of other customers from being mixed into the query text.

[0065] Specifically, the first generation unit 115 adds text data to the query text as conditions, including a description regarding the customer's location information, product information about products located near the customer, a description regarding the customer's characteristic information, a description regarding the customer's product purchase history, and a description regarding the output history of recommended product text.

[0066] As an example, the first generation unit 115 adds text data describing the customer's location information, such as "A (customer's name) is in front of the shelf in the underwear section," to the query text as a condition based on the customer's personal information sent from the first identification unit 112 and the customer's location information sent from the second identification unit 114.

[0067] Furthermore, the first generation unit 115 refers to the nearby product DB 145 and identifies product information corresponding to the customer's location information sent from the second identification unit 114. In this case, it can be said that the first generation unit 115 identifies nearby products in cooperation with the second identification unit 114, and therefore the second identification unit 114 and the first generation unit 115 are an example of a second identification unit.

[0068] The first generating unit 115 adds text data describing the name of the product included in the identified product information as product information of the product located in the vicinity of the customer to the inquiry text as a condition.

[0069] In addition, based on the customer's personal information sent from the first identification unit 112 and the customer's characteristic information sent from the detection unit 113, the first generation unit 115 adds text data describing the customer's characteristic information, such as "A is wearing a white sweater from XX brand and black pants from XX brand," to the query text as a condition.

[0070] In addition, based on the customer's personal information sent from the first identification unit 112, the first generation unit 115 adds text data describing the customer's product purchase history, such as "A purchased a shirt with product code XX on △ month □ day," to the query text as a condition.

[0071] By including a description of the customer's product purchase history as a condition in the inquiry text and adding an instruction to the effect that products that have already been purchased will not be output, it is possible to prevent a situation in which a customer is recommended to purchase a product that the customer has already purchased.

[0072] In addition, based on the customer's personal information sent from the first identification unit 112, the first generation unit 115 adds text data describing the output history of the recommended product text, such as "A has been recommended to purchase the shirt with product code YY ○ times in the past," to the query text as a condition.

[0073] By including in the query text a description of the output history of the recommended product text as a condition and adding to the instructions that products that have not been purchased despite being recommended for purchase a predetermined number of times will not be output, it is possible to prevent situations in which a customer is recommended to purchase a product that has not been purchased despite being recommended for purchase multiple times (a product in which the customer is thought to have little interest) from being recommended for purchase.

[0074] First generation unit 115 associates the generated query text with the customer ID corresponding to the query text and sends it to second generation unit 116. Here, the query text sent to second generation unit 116 includes the time of the customer's location information that is the basis of the generated query text (hereinafter also referred to as the query time).

[0075] Next, the LLM processing unit 102 will be described. The LLM processing unit 102 executes LLM processing. LLM processing is processing in which text is input to LLM and text is obtained as output from LLM. As shown in FIG. 3, the LLM processing unit 102 includes a second generation unit 116 and a fourth generation unit 120. The fourth generation unit 120 will be described later.

[0076] The second generation unit 116 generates recommended product text based on the query text. The recommended product text is an example of second text information. For example, the second generation unit 116 inputs the query text sent from the first generation unit 115 to the promotion LLM 142 stored in the memory unit 14. The second generation unit 116 acquires the text data output from the promotion LLM 142 as the recommended product text.

[0077] The second generation unit 116 also sends the acquired recommended product text to the determination unit 117. Here, the recommended product text sent to the determination unit 117 includes the inquiry time of the inquiry text sent from the first generation unit 115.

[0078] Furthermore, the second generating unit 116 refers to the personal information DB 144 and identifies the output history of the recommended product text corresponding to the customer ID that is sent in association with the inquiry text from the first generating unit 115. The second generating unit 116 performs a process of adding the acquired recommended product text to the output history of the identified recommended product text.

[0079] Note that part or all of the processing of the first generation unit 115 described above may be executed by the second generation unit 116. Also, part or all of the processing of the second generation unit 116 may be executed by the first generation unit 115.

[0080] Next, the LLM post-processing unit 103 will be described. The LLM post-processing unit 103 performs LLM post-processing. For example, the LLM post-processing is a process performed on the recommended product text generated by the second generation unit 116 (promotion LLM 142) in order to generate promotion information in the process of generating promotion information using the promotion LLM 142.

[0081] As shown in FIG. 3, the LLM post-processing unit 103 includes a determination unit 117, a conversion unit 118, a third generation unit 119, a fifth generation unit 121, and an output control unit 122.

[0082] The determination unit 117 makes a determination regarding the output of promotional information. For example, the determination unit 117 determines whether or not to output promotional information based on the customer's location information sent from the second identification unit 114. As an example, the determination unit 117 determines to output promotional information when the customer's location information does not change for more than a predetermined time.

[0083] The determination unit 117 may determine a change in the customer's location information by regarding movement within a predetermined range (for example, movement within a radius of ○ meters of the customer's location information) as no movement.

[0084] Furthermore, when it is determined that the promotion information should be output, the determination unit 117 determines whether a re-inquiry is necessary. For example, the determination unit 117 compares the latest customer location information sent from the second identification unit 114 with the customer location information at the time of the inquiry, and if the two match, determines that a re-inquiry is unnecessary, and if the two do not match, determines that a re-inquiry is necessary.

[0085] In addition, the judgment unit 117 may determine whether or not a re-inquiry is necessary by considering that the location information of both parties matches when the location information changes within a predetermined range (for example, a change within a radius of ○ meters of the customer's location information).

[0086] If the determination unit 117 determines that a re-inquiry is necessary, it sends a re-inquiry request to the first generation unit 115. Furthermore, the determination unit 117 also sends a re-inquiry request in the same manner as above when it determines that promotion information should not be output. Upon receiving the re-inquiry request, the first generation unit 115 generates an inquiry text based on the latest location information, feature information, and personal information, and sends it to the second generation unit 116.

[0087] Furthermore, when the determination unit 117 determines that the promotion information should be output and that a re-inquiry is not necessary, the determination unit 117 determines the output destination of the promotion information. For example, for each of the image display device 2 and the audio output device 3, the determination unit 117 identifies the device (the image display device 2 or the audio output device 3) that is closest to the latest customer location information sent from the second identification unit 114.

[0088] Furthermore, the determination unit 117 determines that the identified image display device 2 and audio output device 3, whichever is closer to the customer's location information, is the output destination of the promotional information. The determination unit 117 also determines whether the other device is within a predetermined range of the latest customer location information. If the other device is within the predetermined range, the determination unit 117 also determines that the other device is the output destination of the promotional information.

[0089] Furthermore, if the devices determined to be the output destination include the image display device 2, the determination unit 117 sends the recommended product text sent from the second generation unit 116 to the conversion unit 118. Here, the recommended product text sent to the conversion unit 118 is associated with information specifying the image display device 2 that is the output destination (hereinafter also referred to as image display device information).

[0090] Furthermore, if the devices determined to be the output destination include the audio output device 3, the determination unit 117 sends the recommended product text sent from the second generation unit 116 to the third generation unit 119. Here, the recommended product text sent to the third generation unit 119 is associated with information specifying the audio output device 3 that is the output destination (hereinafter also referred to as audio output device information).

[0091] The recommended product text sent to the third generating unit 119 is associated with the personal information, characteristic information, and location information of the customer corresponding to the recommended product text.

[0092] In addition, the judgment unit 117 may determine both the image display device 2 closest to the latest customer location information sent from the second identification unit 114 and the audio output device 3 closest to the latest customer location information as the output destination, regardless of the distance from the latest customer location information.

[0093] The conversion unit 118 converts the recommended product text into a promotion image including a product image of the recommended product. The conversion unit 118 is an example of a third generation unit.

[0094] For example, the conversion unit 118 refers to the product image DB 146 and identifies a product image corresponding to the product name included in the recommended product text sent from the determination unit 117. The conversion unit 118 converts the recommended product text into the identified product image. Then, the conversion unit 118 arranges the converted product image according to a predetermined template to generate a promotion image.

[0095] Furthermore, the conversion unit 118 associates the generated promotional image with image display device information and sends it to the output control unit 122.

[0096] The third generation unit 119 generates text for speech conversion based on the recommended product text. Here, the text for speech conversion is instruction information that instructs the speech conversion LLM 143 to generate text.

[0097] For example, the third generation unit 119 generates a text for speech conversion that describes content that instructs the speech conversion LLM 143 to convert the expression of the recommended product text sent from the judgment unit 117 into an expression that does not sound strange when spoken.

[0098] Here, templates for generating instruction information are stored in advance in the memory unit 14 or the like. Note that multiple templates may be stored in the memory unit 14 or the like. In this case, the third generation unit 119 may use different templates depending on the situation. For example, the third generation unit 119 may switch templates depending on keywords (e.g., words such as male and female) included in the text for speech conversion.

[0099] In this embodiment, the instructions for the text to be converted to voice include instructions to convert the text expression, which is the recommended product text to which descriptions regarding the customer's location information, descriptions regarding customer characteristic information, and customer information based on descriptions regarding the customer's personal information (e.g., the customer's current location, physical characteristics, products purchased in the past, etc.), into an expression that does not sound strange when generated as voice.

[0100] Specifically, the third generation unit 119 adds text data including a description about the customer's location information, a description about the customer's characteristic information, and a description about the customer's personal information to the text for speech conversion.

[0101] As an example, based on the customer's personal information and customer location information sent from the judgment unit 117, the third generation unit 119 adds text data describing the customer's location information, such as "Mr. A, browsing the underwear department," to the text for speech conversion.

[0102] Furthermore, based on the customer's personal information and customer characteristic information sent from the judgment unit 117, the third generation unit 119 adds text data describing the customer's characteristic information, such as "Mr. A, who is wearing a white sweater from brand XX," to the text for speech conversion.

[0103] In addition, based on the customer's personal information sent from the judgment unit 117, the third generation unit 119 adds text data describing the customer's personal information, such as "Mr. A, who purchased a shirt with product code XX on △ month □ day," to the text for speech conversion.

[0104] In addition, the instructions in the text for speech conversion may include instructions to specify the output order of each element contained in the recommended product text (for example, the order in which to introduce multiple recommended products), or to summarize and output the contents of the recommended product text.

[0105] Furthermore, the third generation unit 119 associates the generated text for speech conversion with speech output device information and sends it to the fourth generation unit 120.

[0106] Next, the fourth generation unit 120 of the LLM processing unit 102 will be described. The fourth generation unit 120 generates text for promotional voice. The fourth generation unit 120 is an example of the third generation unit. The text for promotional voice is text about recommended products that can be spoken without sounding out of place.

[0107] For example, the fourth generation unit 120 inputs the text for speech conversion sent from the third generation unit 119 to the LLM for speech conversion 143 stored in the memory unit 14. The fourth generation unit 120 acquires the text data output from the LLM for speech conversion 143 as text for promotional speech.

[0108] Furthermore, the fourth generation unit 120 associates the generated promotional voice text with voice output device information and sends it to the fifth generation unit 121.

[0109] Next, the fifth generation unit 121 of the LLM post-processing unit 103 will be described. The fifth generation unit 121 generates a promotional audio. The fifth generation unit 121 is an example of the third generation unit. For example, the fifth generation unit 121 converts the promotional audio text sent from the fourth generation unit 120 into audio data using a known text-to-audio conversion technology, and generates the promotional audio.

[0110] Furthermore, the fifth generation unit 121 associates the generated promotional audio with audio output device information and sends it to the output control unit 122.

[0111] Note that part or all of the processing of the fourth generation unit 120 described above may be executed by the fifth generation unit 121. Also, part or all of the processing of the fifth generation unit 121 may be executed by the fourth generation unit 120.

[0112] The output control unit 122 outputs the promotion information.

[0113] For example, the output control unit 122 transmits the promotional image sent from the conversion unit 118 to the image display device 2 specified by the image display device information associated with the promotional image via the communication line 7. The image display device 2 displays the received promotional image on a display device.

[0114] Furthermore, the output control unit 122 transmits the promotional audio sent from the fifth generation unit 121 to the audio output device 3 specified by the audio output device information associated with the promotional audio via the communication line 7. The audio output device 3 outputs the received promotional audio to the audio output device.

[0115] Next, the processing executed by the SC1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the processing executed by the SC1.

[0116] First, the acquisition unit 111 acquires camera images (step S1). For example, the acquisition unit 111 acquires the camera images by receiving the camera images from each of the multiple cameras 4 in real time via the communication line 7.

[0117] In FIG. 4, the process of acquiring a camera image is described as step S1, but the process of acquiring a camera image is assumed to be performed continuously.

[0118] Next, the first identification unit 112 identifies the personal information of the customer (step S2). For example, the first identification unit 112 extracts a facial image of the customer from the camera image acquired in step S1 using a known image recognition technique. The first identification unit 112 refers to the personal information DB 144 stored in the memory unit 14 and identifies information corresponding to the extracted facial image as personal information.

[0119] In FIG. 4, the process of identifying the customer's personal information is shown as step S2, but the process of identifying the customer's personal information is assumed to be performed continuously.

[0120] Next, the detection unit 113 detects characteristic information of the customer (step S3). For example, the detection unit 113 identifies the customer corresponding to the face image extracted in step S2 from among the customers captured in the camera image acquired in step S1, and detects the external characteristics of the customer using a known image recognition technology.

[0121] In FIG. 4, the process of detecting the characteristic information of the customer is described as step S3, but the process of detecting the characteristic information of the customer is assumed to be performed continuously.

[0122] Next, the second identification unit 114 identifies the location information of the customer (step S4). For example, the second identification unit 114 receives a reception result of radio waves including a terminal ID from the mobile terminal 6 via the communication line 7 (including the terminal ID of the mobile terminal 6).

[0123] The second identification unit 114 identifies the location within the store of the mobile terminal 6 of the customer, based on the reception result including the terminal ID of the mobile terminal 6 of the customer, which is included in the personal information of the customer identified in step S2. The second identification unit 114 identifies the identified location of the mobile terminal 6 as location information of the customer.

[0124] In FIG. 4, the process of identifying the customer's location information is shown as step S4, but the process of identifying the customer's location information is assumed to be performed continuously.

[0125] Next, first generation unit 115 generates a query text (step S5). For example, first generation unit 115 generates a query text instructing the generation of text about products available at the store, with a description about the personal information of the customer identified in step S2, a description about the characteristic information of the customer detected in step S3, and a description about the location information of the customer identified in step S4 added as conditions.

[0126] Next, the second generation unit 116 generates recommended product text (step S6). For example, the second generation unit 116 inputs the inquiry text generated in step S5 to the promotion LLM 142 stored in the memory unit 14. The second generation unit 116 acquires the text data output from the promotion LLM 142 as the recommended product text.

[0127] Next, the determination unit 117 determines whether or not to output promotion information (step S7).

[0128] For example, the determination unit 117 determines to output promotion information when the location information of the customer continuously identified in step S4 does not change for a predetermined period of time. On the other hand, when the location information of the customer changes, the determination unit 117 determines not to output promotion information. When it is determined not to output promotion information (step S7: No), the process proceeds to step S9 described below.

[0129] On the other hand, if it is determined that the promotion information should be output (step S7: Yes), the determination unit 117 determines whether or not another inquiry is necessary (step S8).

[0130] For example, the determination unit 117 compares the customer location information corresponding to the inquiry text generated in step S5 with the latest location information, and if the two do not match, determines that a re-inquiry is necessary. On the other hand, if the two match, the determination unit 117 determines that a re-inquiry is not necessary.

[0131] If it is determined that a re-inquiry is necessary (step S8: Yes), the determination unit 117 sends a re-inquiry request to the first generation unit 115 (step S9), and proceeds to the processing of step S5. Upon receiving the re-inquiry request, the first generation unit 115 executes the processing of step S5 based on the latest personal information, characteristic information, and location information of the customer acquired in the processing of steps S1 to S4 that are continuously executed.

[0132] On the other hand, if it is determined that a re-inquiry is not necessary (step S8: No), the determination unit 117 determines whether the image display device 2 is included in the output destinations (step S10).

[0133] For example, if the image display device 2 closest to the latest customer location information identified in step S4 is closer to the latest customer location information than the audio output device 3 closest to the latest customer location information, the judgment unit 117 determines that the image display device 2 is included in the output destinations.

[0134] Furthermore, even if the audio output device 3 is closer to the latest customer location information, if the image display device 2 closest to the latest customer location information is within a predetermined range of the latest customer location information, the determination unit 117 determines that the image display device 2 is included in the output destinations. If the image display device 2 is not included in the output destinations (step S10: No), the process proceeds to step S13, which will be described later.

[0135] On the other hand, if the image display device 2 is included in the output destination (step S10: Yes), the conversion unit 118 converts the recommended product text into a promotion image (step S11).

[0136] For example, the conversion unit 118 refers to the product image DB 146 stored in the memory unit 14 and identifies a product image corresponding to the product name included in the recommended product text. The conversion unit 118 converts the recommended product text into the identified product image, and generates a promotion image by arranging the converted image according to a template.

[0137] Next, the output control unit 122 transmits the promotional image to the image display device 2 (step S12). For example, the output control unit 122 transmits the promotional image converted in step S11 via the communication line 7 to the image display device 2 that is closest to the latest customer location information identified in step S4.

[0138] Next, the determination unit 117 determines whether the output destinations include the audio output device 3 (step S13).

[0139] For example, if the audio output device 3 closest to the latest customer location information identified in step S4 is closer to the latest customer location information than the image display device 2 closest to the latest customer location information, the judgment unit 117 determines that the audio output device 3 is included in the output destinations.

[0140] Furthermore, even if the image display device 2 is closer to the latest customer location information, if the audio output device 3 closest to the latest customer location information is within a predetermined range of the latest customer location information, the determination unit 117 determines that the audio output device 3 is included in the output destinations. If the audio output device 3 is not included in the output destinations (step S13: No), this process ends.

[0141] On the other hand, if the output destination includes the voice output device 3 (step S13: Yes), the third generation unit 119 generates a text for voice conversion (step S14).

[0142] For example, the third generation unit 119 generates a text for speech conversion that instructs the generation of a text for promoting a product by adding a description based on the customer's personal information, customer characteristic information, and customer location information corresponding to the recommended product text generated in step S6.

[0143] Next, the fourth generating unit 120 generates a promotional voice text (step S15).

[0144] For example, the fourth generation unit 120 inputs the text for speech conversion generated in step S15 to the LLM for speech conversion 143 stored in the memory unit 14. The third generation unit 119 acquires the text data output from the LLM for speech conversion 143 as the text for promotional speech.

[0145] Next, the fifth generation unit 121 generates a promotional audio (step S16). For example, the fifth generation unit 121 converts the promotional audio text generated (acquired) in step S15 into audio data using a known technology for converting text data into audio data, thereby generating the promotional audio.

[0146] Next, the output control unit 122 transmits the promotional sound to the sound output device 3 (step S17). For example, the output control unit 122 transmits the promotional sound generated in step S16 via the communication line 7 to the sound output device 3 that is closest to the latest customer location information identified in step S4.

[0147] 4, the process of step S13 is described as a process after step S12, but the process of step S13 may be executed in parallel with step S10. Moreover, the process of step S13 may be executed after the process of step S8 and before step S10.

[0148] As described above, the SC1 in this embodiment acquires camera images from a camera installed in the facility, detects the characteristics of customers captured in the camera images, and generates recommended product text by inputting an inquiry text that expresses the detected customer characteristics into the promotional LLM142, which outputs text about products handled in the store in response to input of inquiry text that includes the person's characteristics, and outputs information based on the recommended product text.

[0149] As a result, the SC1 according to this embodiment can generate an inquiry text that takes into account the detected customer characteristics. By inputting the inquiry text that takes into account the customer characteristics into the promotion LLM 142 and generating a recommended product text, it is possible to generate a recommended product text that is more optimized for the customer. In other words, the SC1 according to this embodiment can improve the convenience of promotions for customers that use the generation AI.

[0150] Furthermore, the SC1 according to this embodiment generates an inquiry text that includes a description of the customer's appearance as a condition, which allows the SC1 according to this embodiment to generate a recommendation text that takes into account the customer's current clothing, emotions that can be inferred from the customer's facial expression, and so on.

[0151] Furthermore, the SC1 according to this embodiment identifies the customer from the camera image, identifies the customer's product purchase history from the personal information DB 144 that includes the product purchase history, and generates an inquiry text with a description of the identified product purchase history added as a condition. This makes it possible for the SC1 according to this embodiment to prevent a situation in which a product that has already been purchased is recommended to the customer.

[0152] Furthermore, the SC1 according to this embodiment identifies the customer's location information within the store, identifies product information of products located near the identified location information from the identified location information, and generates an inquiry text with a description of the identified product information added as a condition. As a result, the SC1 according to this embodiment can recommend the purchase of products near the customer's current location.

[0153] Furthermore, the SC1 according to this embodiment converts the recommended product text, which includes the product name, into a product image based on the product image DB 146, which associates product names with product images, and generates a promotional image that includes the product image and is to be output to the image display device 2. This makes it easier for customers to understand what products they are being recommended to purchase. In other words, the SC1 according to this embodiment can carry out promotions in a way that is visually easy for customers to understand.

[0154] Furthermore, the SC1 according to this embodiment converts the recommended product text, which includes the product name, into promotional voice text based on the recommended product text and the voice conversion LLM 143, which converts the recommended product text into promotional voice text that sounds natural as voice, and then converts the promotional voice text into voice data to generate promotional voice for output to the voice output device 3. This allows customers to understand the products they are being recommended to purchase even in situations where they cannot see product images. In other words, the SC1 according to this embodiment can promote products in a way that appeals to the customer's hearing.

[0155] Furthermore, the SC1 according to this embodiment outputs promotional information to the output device when the change in the customer's location information within a predetermined time period is within a predetermined range. Here, no (little) change in the location information means that the customer is stopping at the location within the store represented by the location information. Also, stopping often indicates that the customer is interested in that location. In other words, the SC1 according to this embodiment outputs promotional information related to the location information only when there is no (little) change in the location information, thereby preventing situations in which the customer is encouraged to purchase products that they are not interested in.

[0156] Furthermore, the SC1 according to this embodiment outputs promotional information to an output device that is close to the customer's current location, thereby preventing the SC1 according to this embodiment from outputting promotional information in a manner that is not conveyed to the customer.

[0157] The above-described embodiment can be modified as needed by changing some of the configurations or functions of SC1. Therefore, below, several modifications of the above-described embodiment will be described as other embodiments. Below, differences from the above-described embodiment will be mainly described, and detailed descriptions of commonalities with the contents already described will be omitted. The modifications described below may be implemented individually or in appropriate combination.

[0158] (Variation 1) In the above-described embodiment, a configuration has been described in which SC1 includes the promotion LLM 142 and the voice conversion LLM 143. In this modification, a configuration will be described in which a server device other than SC1 includes the promotion LLM 142 and the voice conversion LLM 143.

[0159] In this modification, a server device (hereinafter simply referred to as the server device) equipped with a promotion LLM 142 and a voice conversion LLM 143 is installed inside the store, similar to SC1. The server device may be installed outside the store, but because information including personal information of customers is input into the promotion LLM 142 and the voice conversion LLM 143, if the server device is installed outside the store, it is preferable to install it in a location that cannot be easily accessed by persons unrelated to the store.

[0160] In this modified example, the second generation unit 116 transmits the inquiry text sent from the first generation unit 115 to the server device via the communication line 7. The server device inputs the received inquiry text to the promotion LLM 142. The server device transmits the recommended product text output from the promotion LLM 142 to SC1. The second generation unit 116 transmits the recommended product text sent from the server device to the determination unit 117.

[0161] Furthermore, the fourth generation unit 120 transmits the text for speech conversion generated by the third generation unit 119 to the server device via the communication line 7. The server device inputs the received text for speech conversion to the speech conversion LLM 143. The server device transmits the text for promotional speech output from the speech conversion LLM 143 to SC1. The fifth generation unit 121 converts the text for promotional speech transmitted from the server device into promotional speech.

[0162] In the above, the promotion LLM 142 and the voice conversion LLM 143 are provided on the same server device, but the server device providing the promotion LLM 142 and the server device providing the voice conversion LLM 143 may be different server devices.

[0163] According to this modification, the processing load on SC1 can be reduced.

[0164] (Variation 2) In the above-described embodiment, the information processing system S is described as including both the image display device 2 and the audio output device 3. However, the information processing system S may be provided with only one of the image display device 2 and the audio output device 3.

[0165] According to this modification, for example, promotions can be carried out to customers using only an output device suited to the circumstances of the store.

[0166] (Variation 3) In the above-described embodiment, a configuration has been described in which a customer's product purchase history is stored in the personal information DB 144 of the memory unit 14 of the SC1. In this modified example, a configuration in which a customer's product purchase history is stored in the customer's mobile terminal 6 will be described.

[0167] In this modification, the first identification unit 112 identifies the terminal ID of the customer's mobile terminal 6 from the personal information DB 144. The first identification unit 112 acquires the customer's product purchase history from the mobile terminal 6 identified by the identified terminal ID via the access point 5 and the communication line 7.

[0168] According to this modification, it is possible to reduce the amount of information stored in the personal information DB 144 of the memory unit 14 of the SC1.

[0169] (Variation 4) In the above embodiment, the SC1 converts the recommended product text into image data or audio data and outputs it. However, the SC1 may output the recommended product text to the display device as text data.

[0170] According to this modification, the recommended product text can be displayed on a simple display device that can display only text, so that product promotion can be carried out at low cost.

[0171] The program executed by SC1 in the above-described embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0172] The program executed by the SC1 of the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the SC1 of the embodiment may be provided or distributed via a network such as the Internet.

[0173] Furthermore, the program executed by the SC1 of the embodiment may be provided by being pre-installed in the ROM 12 or the like.

[0174] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents set forth in the claims. [Explanation of symbols]

[0175] 1 SC 2 Image display device 3 Audio output devices 4. Camera 5. Access Points 6. Mobile devices 7. Communication lines 11 CPU 12 ROM 13 RAM 14 Memory section 100 control section 101 LLM Pre-Processing Section 102 LLM Processing Section 103 LLM Post-Processing Unit 111 Acquisition Department 112 1st Specific Part 113 Detection unit 114 Second Specific Part 115 1st generation part 116 Second generation part 117 Judgment Department 118 Conversion Unit 119 Third generation part 120 4th generation part 121 5th generation part 122 Output control section 142 Promotional LLM 143 LLM for Voice Conversion 144 Personal information DB 145 Nearby Product DB 146 Product Image DB [Prior art documents] [Patent documents]

[0176] [Patent Document 1] Japanese Patent Publication No. 2023-098483

Claims

1. an acquisition unit that acquires camera images captured by cameras installed in the facility; a detection unit that detects features of a customer captured in the camera image; a first generation unit that generates first text information representing the characteristics of the customer based on the detected characteristics; a second generation unit that inputs the inquiry text to which the first text information has been added to a generation AI that generates text related to products handled at the facility in response to the input of the inquiry text, and generates second text information related to product promotion based on the text generated by the generation AI; an output control unit that outputs the second text information; An information processing device comprising:

2. the detection unit detects the appearance characteristics of the customer; the first generation unit generates the first text information representing the detected appearance characteristics of the customer; the second generation unit inputs the query text, to which a description of appearance characteristics is added as the condition, into a first large-scale language model, which is the generation AI and is functionalized to output text related to a product based on an input of text specifying a condition, and acquires the text output by the first large-scale language model as the second text information; The information processing device according to claim 1 .

3. a second acquisition unit that acquires a purchase history of a customer corresponding to the detected characteristic; the second generation unit inputs the inquiry text to which the acquired description regarding the purchase history is added as the condition into the first large-scale language model, and acquires text output by the first large-scale language model as the second text information; The information processing device according to claim 2 .

4. the second acquisition unit acquires the purchase history from a database that stores the characteristics of each customer and the purchase history of the customer in association with each other, or from a mobile terminal carried by the customer; The information processing device according to claim 3 .

5. a second identification unit that identifies nearby products located near the customer based on location information representing a location of the customer within the facility; the second generation unit inputs the query text to which a description about the identified nearby product is added as the condition to the first large-scale language model, and acquires text output by the first large-scale language model as the second text information; The information processing device according to claim 2 .

6. a third generation unit that generates promotional information by converting the content expressed in the second text information into data including at least one of image data and audio data; the output control unit outputs the generated promotion information to an output device. The information processing device according to claim 5 .

7. the third generation unit inputs the second text information into a second large-scale language model, which is a generation AI that is functionalized to output text for promoting a product in response to input of text related to the product, and generates the promotion information by converting the text output by the second large-scale language model into voice data; The information processing device according to claim 6 .

8. the third generation unit reads out the image data corresponding to the name of the product included in the second text information from a product image database in which the name of each product is associated with image data of the product, and generates the promotion information based on the read image data and the second text information. The information processing device according to claim 6 .

9. the output control unit outputs the promotion information to the output device when a change in the location information of the customer within a predetermined time period is within a predetermined range. The information processing device according to claim 6 .

10. the output control unit outputs the promotion information to the output device that is closest to the latest location information of the customer. The information processing device according to claim 6 .

11. An information processing method by an information processing device, Acquire camera images captured by cameras installed in the facility; Detecting features of the customer captured in the camera image; generating first text information representing the customer's characteristics based on the detected characteristics; inputting the inquiry text to which the first text information has been added to a generation AI that generates text related to products handled at the facility in response to the input of the inquiry text, and generating second text information related to product promotions based on the text generated by the generation AI; outputting the second text information; An information processing method including the steps.

Images