Information processing apparatus, information processing method, and program

Abstract

To effectively communicate information about products to customers in a store.SOLUTION: An information processing apparatus acquires an image of the interior of a store 100 from an image capture unit 101. The apparatus analyses the behavior of people in the image. On the basis of position information of products displayed in the store 100 and the behavior analysis results, the apparatus estimates target products 105. Then, on the basis of information of the target product 105 and the behavior analysis results, the apparatus generates text information to be communicated to the person in the image. The apparatus then controls the conversion of the generated text information into voice data and the output of the data from a voice output unit 102.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for analyzing and processing an image. [Background technology]

[0002] Conventionally, for the purpose of marketing and sales promotion, various facilities such as retail stores and large commercial stores operate systems that analyze customer behavior and attributes based on images taken by cameras installed in the store and advertise product information to customers based on the analysis results. Patent Document 1 discloses a system that selects promotional content based on customer behavior and attributes acquired from cameras or the like installed on product shelves, and displays the selected promotional content on displays installed on the product shelves. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-72337 Summary of the Invention [Problem to be solved by the invention]

[0004] In the system of Patent Document 1, promotional content shown on displays installed on product shelves must be created in advance according to customer behavior and attributes, making it difficult to increase the variety. Also, customers must look away from the products to check the displays, which may prevent product information from being provided to customers in a timely manner.

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to effectively convey information about products to customers in a store. [Means for solving the problem]

[0006] The present invention is characterized by having an image acquisition means for acquiring an image from an imaging unit that images the inside of a store, a behavior analysis means for analyzing the behavior of people in the image, a product estimation means for performing an estimation process of a target product based on position information of the product in the store and the analysis results by the behavior analysis means, a text generation means for performing a generation process of generating text information to be communicated to the people in the image based on product information of the target product and the analysis results by the behavior analysis means, and an output control means for controlling the generated text information to be converted into audio data and output. Effect of the Invention

[0007] According to the present invention, information about products can be effectively communicated to customers in a store. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a store system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] 1 is a diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment. [Figure 4] 4 is a flowchart showing a process of the information processing device according to the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing device according to a second embodiment. [Figure 6] 10 is a flowchart showing a process of an information processing device according to a second embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing device according to a third embodiment. [Figure 8] 11 is a flowchart showing a process of an information processing device according to a third embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of a functional configuration of an information processing device according to a fourth embodiment. [Figure 10] 13 is a flowchart showing a process of an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] <Embodiment 1> Fig. 1 shows the overall configuration of a store system according to this embodiment. A store 100 is provided with an imaging unit 101 as an imaging device, an audio output unit 102 that transmits information to a customer 105 in the store 100, and product shelves 103 and 104. Fig. 1 illustrates a scene in which a customer 105 picks up a product 110 displayed on the product shelf 104, and the imaging unit 101 captures the customer's action. The imaging unit 101 and the audio output unit 102 are connected to an information processing device 300 and controlled by the information processing device 300.

[0011] The store 100 may be a supermarket, a convenience store, a drug store, an apparel store, or the like, and is not particularly limited as long as it sells products to customers. The information processing device 300 is connected to an external storage device 222. The external storage device 222 stores product information including the positions and types of products displayed in the store 100. Furthermore, a display unit 351 is connected to the information processing device 300 for presenting information such as the status of the information processing device 300 and the store system to employees of the store 100.

[0012] 2 shows a hardware configuration of the information processing device 300. The information processing device 300 includes a CPU (Central Processing Unit) 201 that functions as a control unit (arithmetic unit). The information processing device 300 also includes a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and a HDD (Hard Disk Drive) 204 as storage units. The information processing device 300 also includes a recording disk drive 205 and various interfaces 206 to 209. The information processing device 300 is a general computer. The CPU 201, the ROM 202, the RAM 203, the HDD 204, the recording disk drive 205, and the various interfaces 206 to 209 are connected via a bus 210. The various interfaces 206 to 209 are connected to various devices including an imaging unit 101 and an audio output unit 102.

[0013] The CPU 201 controls the entire information processing device 300. The ROM 202 stores basic programs such as BIOS. The RAM 203 temporarily stores various data such as the results of calculations by the CPU 201. The HDD 204 stores the results of calculations by the CPU 201 and various data acquired from the outside, and records various programs 204a and various parameters 204b executed by the CPU 201. The CPU 201 executes the processing of each step of a flowchart described below based on the various programs 204a and various parameters 204b stored in the HDD 204.

[0014] The various programs 204a refer to various programs necessary for each step of the flowchart described later, such as a control program for each device, a posture estimation program for a person in an image, a behavior analysis program, a text generation program, an information transmission instruction program, etc. Also, the various parameters 204b refer to various parameters necessary for each step of the flowchart described later, such as a control parameter for each device, an image processing parameter for posture estimation, a parameter for behavior determination, and a model parameter obtained by machine learning for text generation.

[0015] In the first embodiment, a case will be described in which the storage medium that the information processing device 300 can read and write is the HDD 204, and the various programs 204a and the various parameters 204b are stored in the HDD 204, but the present invention is not limited to this. The various programs 204a and the various parameters 204b may be stored in any storage medium as long as they are stored in a storage medium that the information processing device 300 can read, and the ROM 202, the recording disk 231, the external storage device 222, etc. shown in Fig. 2 may be used. Specific configurations of the storage medium may include a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory, a ROM, etc.

[0016] The imaging unit 101 is a device having a solid-state imaging element that receives light and generates image data, specifically a digital camera, and is controlled by an instruction output by the CPU 201. The solid-state imaging element is an area sensor such as a CMOS image sensor or a CCD image sensor. The imaging unit 101 is connected to the information processing device 300 via an interface 206. The captured image data is output to the CPU 201 via the interface 206. The imaging unit 101 may also be a surveillance camera installed from the ceiling of the store 100 so as to look down on the inside of the store 100. The imaging unit 101 may be installed at a position where it can capture an image of the front of each product shelf where products are displayed. There is no particular limitation on the position as long as it can capture an image that allows the state of customers in the store to be analyzed.

[0017] The audio output unit 102 is specifically a speaker, and is controlled by commands output by the CPU 201. The audio output unit 102 is connected to the information processing device 300 via an interface 209. The CPU 201 transmits the generated audio signal to the audio output unit 102 via the interface 209. As a result, the audio output unit 102 conveys information about the product to the customer 105 as audio.

[0018] 1, one each of the imaging unit 101 and the audio output unit 102 is shown, but there may be at least one of these in the store system, and there may be multiple of them. The number of imaging units 101 and audio output units 102 installed in the store system may vary from store to store depending on the size of the store, etc., and may be changed as appropriate depending on the situation.

[0019] The recording disk drive 205 is a device capable of reading various data, programs, and the like recorded on a recording disk 231 . The display unit 351 is a monitor using a liquid crystal panel, an organic EL panel, or the like, and is controlled by commands output by the CPU 201. The display unit 351 is connected to the information processing device 300 via the interface 207. Information to be displayed is generated by the CPU 201 and output to the display unit 351 via the interface 207. As a result, the display unit 351 displays characters and images indicating the processing results and operating status of the information processing device 300, the status of the store system, and the like.

[0020] The external storage device 222 stores data used by the information processing device 300. The external storage device 222 is connected to the information processing device 300 via the interface 208. Examples of information stored in the external storage device 222 include product information of products displayed in the store 100 and various information used in each step of the flowchart described later. Although only one external storage device 222 is shown in FIG. 1, the number of external storage devices is not limited to one, and may be multiple. The location of the external storage device 222 may be inside or outside the store, or may be inside the information processing device 300. In other words, there is no particular limitation as long as the location is accessible from the information processing device 300. The information processing device 300 may also include a network interface (not shown). The information processing device 300 may be connected to the Internet via the network interface to communicate with a customer's smartphone, an external server, or the like.

[0021] Fig. 3 shows a functional configuration of an information processing device 300 according to this embodiment. The CPU 201 of the information processing device 300 reads out and operates various programs 204a and various parameters 204b, thereby implementing the functions of the respective functional units 321 to 329 shown in Fig. 3. The information processing device 300 includes an image acquisition unit 321 for acquiring image data from the imaging unit 101. The information processing device 300 further includes a posture estimation unit 322 for estimating a posture of the customer 105 captured in the image acquired by the image acquisition unit 321 and obtaining posture estimation information.

[0022] Here, the posture estimation information includes information such as the position of the customer 105 on the image, the position of each joint point, and their likelihood (score value), which are obtained by image analysis of the image acquired by the image acquisition unit 321. The position of the customer 105 on the image is, for example, information on the center coordinates of a rectangle surrounding the customer on the image, with the upper left corner of the image as the origin, and the width and height of the rectangle. Also, the position of each joint point of the customer 105 is the center coordinates of each joint of the customer 105 when the upper left corner of the image is the origin, as described above. The posture estimation information is stored in chronological order in the RAM 203, etc. Note that the posture estimation information may also include information on the facial expression and gaze of the customer 105.

[0023] Furthermore, the information processing device 300 has a behavior memory unit 324 and a behavior determination unit 323. The behavior determination unit 323 analyzes behavior information of the customer 105 based on the time-series posture estimation information of the customer 105 obtained by the posture estimation unit 322, and determines whether the customer 105 has performed a behavior stored in the behavior memory unit 324. Here, the behavior is, for example, a motion sequence that represents a time-series change in a person's posture.

[0024] In this embodiment, the behavior determination unit 323 determines whether the customer 105 is interested in a product by comparing the behavior information of the customer 105 with the behavior stored in the behavior memory unit 324. For this purpose, the behavior memory unit 324 stores a pattern of behavior that the customer takes when he or she is interested in a product. Here, various behaviors that indicate that the customer 105 is interested in a product are assumed. For example, as shown in FIG. 1, the behavior of the customer 105 picking up a product on the product shelf 104 is assumed. In this case, the behavior determination unit 323 detects each joint point of the person from the image, and determines whether or not there is a series of behavior sequences in which an arm is stretched to the product shelf and then the arm is retracted from the time series changes of each detected joint point. If the behavior sequence is present, the behavior determination unit 323 determines that the customer 105 is interested in a product displayed on the product shelf 104.

[0025] In addition to the above, examples of behaviors that indicate that the customer 105 is interested in a product include a behavior in which the customer 105 stops in front of the product shelf 104 and looks at the products displayed thereon, and a change in the customer 105's facial expression or gaze. The behavior determination unit 323 may also determine whether the customer 105 is interested in a product based on behavioral information of the customer 105 after transmitting information about the product to the customer 105 by voice. In this case, examples of behaviors that indicate that the customer 105 is interested in a product include a behavior in which the customer 105 puts a product in a shopping basket or cart, and a change in the customer 105's facial expression or gaze after putting the product in a shopping basket or cart. A pattern of behaviors that indicate that the customer 105 is interested in a product may be used in combination with a plurality of patterns. Hereinafter, a description will be given assuming that a behavior that indicates that the customer 105 is interested in a product is a behavior in which the customer 105 picks up a product from the product shelf 104 as shown in FIG. 1.

[0026] The information processing device 300 further includes a product information acquisition unit 325. The product information acquisition unit 325 acquires product information including the position and type of products displayed on the product shelf 104 from the product information data 340. The product information data 340 is a data group of product information, and is stored in the external storage device 222. The product information refers to all information related to products sold in the store 100, such as position information indicating which product shelf and which position the product is displayed on in the store 100, information related to the performance and quality of the product, and discount information for the product. The product information may further include electronic coupons distributed in advance via an application of the store 100 installed on the smartphone of the customer 105, and discount information posted on the homepage of the store 100. The product information may also include information indicating who the celebrity is who will be advertising the product in a television commercial for the product.

[0027] Here, it is assumed that the product information data 340 is a database organized into tables for each product, but the format is not limited to a database and is not particularly limited as long as the format allows product information to be extracted and updated. Also, here, the product information data 340 is in a format in which it is aggregated in one place, but it may be in a format in which it is distributed in multiple places.

[0028] Furthermore, the information processing device 300 has a product estimation unit 326. The product estimation unit 326 estimates which product the customer 105 has picked up, based on the position information of the products displayed on the product shelf 104 obtained by the product information acquisition unit 325 and the behavior information of the customer 105 obtained by the behavior determination unit 323.

[0029] The information processing device 300 further includes a text generation unit 327. The text generation unit 327 automatically generates text using a text generation model 350 that has been trained in advance using a deep learning technique. The text generation model 350 is trained to output text for promoting the purchase of a product using behavioral information of a person in a store and product information of the product as input information. A machine learning algorithm such as a recurrent neural network (RNN) is used as the text generation model 350. The text generation unit 327 generates text to be communicated to the customer 105 by inputting behavioral information of the customer 105 and product information of the product 110 that is estimated to have been picked up by the customer 105 into the text generation model 350. The text generated here is, for example, text information related to discount information on the product 110 and product information of another product that is discounted when purchased as a set with the product 110. In addition, the method is not limited to using the sentence generation model 350, but a sentence for promoting the purchase of the product 110 may be generated based on the behavioral pattern of the customer 105 and the estimated product information of the product 110 by performing rule-based processing such as LUT.

[0030] A sentence generated using the sentence generation model 350 changes according to behavioral information of the customer 105 input to the sentence generation model 350. For example, the sentence changes according to the behavior of the customer 105 picking up a product on the product shelf 104, the behavior of the customer 105 stopping in front of the product shelf 104 and looking at the products displayed, the behavior of the customer 105 putting a product in a shopping basket or cart, etc. Specifically, from the behavior of the customer 105 stopping in front of the product shelf 104 and looking at the products displayed, a sentence including information on the performance and quality of the product that introduces the product at the customer's line of sight is generated. Also, from the behavior of the customer 105 picking up a product on the product shelf 104, a sentence including discount information that leads to the purchase of the product is generated. Also, from the behavior of the customer 105 putting a product in the shopping basket or cart, a sentence including information on another product that is discounted when purchased as a set with the product is generated.

[0031] Furthermore, the information processing device 300 has an information transmission instruction unit 328. The information transmission instruction unit 328 converts the sentence generated by the sentence generation unit 327 into a voice signal, and transmits the converted voice signal to the voice output unit 102. As a result, the voice output unit 102, i.e., the speaker, transmits the sentence to be transmitted to the customer 105 by voice.

[0032] Here, if there are multiple audio output units in the store system, the information transmission instruction unit 328 may select an audio output unit whose audio is most likely to reach the customer 105 from among the multiple audio output units. Specifically, the position of the customer 105 in the store 100 is estimated using the estimation result of the posture estimation unit 322 for the customer 105 in the captured image, and an audio output unit to which the audio signal is to be transmitted is selected based on the position of the customer 105 and the respective positions of the multiple audio output units. Here, setting parameters for selecting the audio output unit are set in advance by a person in charge of the store 100, stored in a storage medium accessible by the information processing device 300, such as the HDD 204 or the external storage device 222, and used when the store system is in operation. Also, if there are multiple imaging units in the store system, the information transmission instruction unit 328 may select an audio output unit to which the audio signal is to be transmitted based on the position of the imaging unit at which the customer 105 was captured.

[0033] Furthermore, the information transmission instruction unit 328 may freely change the voice for reading the text by using a voice synthesis technique. For example, the information transmission instruction unit 328 may transmit to the voice output unit 102 a voice signal changed for each product estimated by the product estimation unit 326. The information transmission instruction unit 328 may also determine the type of voice itself based on the product information stored in the product information data 340. The information transmission instruction unit 328 may also transmit text information and the type of voice corresponding to the product estimated by the product estimation unit 326 to the voice output unit 102. In this case, the voice output unit 102 generates and outputs a voice signal according to the instructed type of voice from the received text information. The information processing device 300 further includes a display instruction unit 329. The display instruction unit 329 displays information such as the status of the information processing device 300 and the store system on a display unit 351, and presents the information to an employee of the store 100, who is a user of the store system.

[0034] 4 is a flowchart showing an example of the flow of processing executed by the information processing device 300 according to this embodiment. In the following description, each process (step) is represented by adding an S to the beginning thereof, and the representation of the process (step) is omitted. In S401, the image acquisition unit 321 captures images of the inside of the store 100 using the imaging unit 101, and acquires image data from the imaging unit 101. The image data acquired by the image acquisition unit 321 may be a moving image or a still image. In the case of a moving image, a frame image is treated as one image.

[0035] In S402, the posture estimation unit 322 estimates the posture of the customer 105 from the image acquired in S401 to acquire posture estimation information, and stores it in the RAM 203 or the like. In S403, the behavior determination unit 323 analyzes behavior information of the customer 105 based on the time series data of the posture estimation information of the customer 105 stored in S402. The process of this step is an example of a behavior analysis process that analyzes the behavior of a person in an image. The process of this step provides a behavior analysis result.

[0036] In S404, the behavior determination unit 323 determines whether or not the customer 105 has shown an interest in the product by comparing the behavior information of the customer 105 analyzed in S403 with the behavior pattern stored in the behavior storage unit 324. If the behavior determination unit 323 determines that the customer 105 has shown an interest in the product, the process proceeds to S405, and if it determines that the customer has not shown an interest in the product, the process returns to S401 to take a picture again. By executing the processes of S401 to S403 again, the information processing device 300 analyzes the next movement of the customer 105. In S405, the product information acquisition unit 325 acquires product information of the products displayed in the store 100 from the product information data 340. Then, the product estimation unit 326 estimates the product (target product 110) in which the customer 105 shows interest, based on the position information of the products displayed in the store 100 and the behavior information of the customer 105 analyzed in S403. The process of the product estimation unit 326 in this step is an example of a process of estimating the target product.

[0037] In S406, the text generation unit 327 inputs the product information of the target product 110 and the behavior information of the customer 105 analyzed in S403 into the text generation model 350 to generate text (text data) to be communicated to the customer 105. The processing in this step is an example of a generation process that generates text information to encourage a person in an image to purchase the target product. In S407, the information transmission instruction unit 328 converts the text generated in S406 into an audio signal for output from the audio output unit 102, and transmits it to the audio output unit 102. The audio output unit 102 receives and outputs the audio signal from the information transmission instruction unit 328, whereby information about the target product 110 is transmitted by voice to the customer 105 in the store 100. The processing of the information transmission instruction unit 328 in this step is an example of processing for converting the generated text information into audio data and outputting it. Thereafter, the series of steps in the flowchart ends.

[0038] The information processing device 300 continuously acquires images of the customer 105 and repeatedly executes a series of steps in the flowchart at a predetermined interval. This makes it possible to monitor the movement of the customer 105 and, at the timing when a movement indicating an interest in a product is detected, to transmit to the customer 105 by voice, purchase promotion information for the product that is estimated to be of interest.

[0039] According to the first embodiment, since texts advertising products to customers are automatically generated according to customer behavior and product type, the system implementation cost and implementation period can be reduced compared to the case where pre-created display content is selected according to customer behavior and product type. Also, information for advertising products can be transmitted to customers in various variations. Also, by reading out the automatically generated text aloud, information for advertising products can be transmitted to a wider range of areas in the store, so the cost of installing devices can be reduced compared to the case where products are promoted on a display. Also, by transmitting information by voice, customers do not need to take their eyes off the products, so information for advertising products can be input to customers in a timely manner. As a result, customers' purchasing motivation is less likely to decrease, the probability of purchasing products is increased, and an increase in the average customer spending can be expected.

[0040] <Embodiment 2> In the store system according to the second embodiment, in addition to customer behavior information and product information, attribute information of customers who have shown interest in the products is used to generate text to be communicated to customers. In the following, the same parts as in the first embodiment will be omitted, and the differences from the first embodiment will be mainly described.

[0041] Fig. 5 shows a functional configuration of an information processing device 300 according to this embodiment. The information processing device 300 in Fig. 5 differs from the functional configuration of the information processing device 300 in Fig. 3 in that a customer attribute estimation unit 501 and an attribute-based product purchase analysis unit 502 are added. The customer attribute estimation unit 501 performs image analysis on the image of the customer 105 in the store 100 acquired by the image acquisition unit 321, and estimates the attributes of the customer 105. Here, the attributes are, for example, age, sex, body type, hairstyle, and clothing, and may be information that combines these, and are not particularly limited as long as the information can classify the customer category. For example, a category that combines multiple attributes, such as a certain age and a specific sex, may be defined, and a category that combines multiple combinations of multiple attributes may be defined.

[0042] Moreover, the information processing device 300 according to this embodiment uses attribute-specific product purchase information data 510. The attribute-specific product purchase information data 510 is a data group in which attribute information of a customer who purchased a product is added to sales information of the product at the store 100 (such as the product name, sale date and time, and sales quantity of the product sold), and is stored in the external storage device 222. A method of acquiring the customer attribute information to be added to the product sales information will be described below.

[0043] For example, there is a method in which a store employee in charge of a cash register looks at a customer, determines the customer's attributes, inputs the customer's attribute information from a cash register terminal each time a transaction is made, and adds the input attribute information to the sales information of the product purchased by the customer and stores it. There are other methods as follows. A method in which an application that is pre-installed on a smartphone owned by a customer is used to accumulate discount points according to the purchase price when purchasing a product. In this application, a customer ID is assigned to each customer. When this application is started for the first time, the customer's attribute information, such as the customer's date of birth, age, and sex, entered by the customer himself is linked to the customer ID and stored in a server managed by the store. When a customer makes a transaction at a cash register, the customer starts the above-mentioned application and presents the screen of the smartphone, which displays a barcode or QR code (registered trademark) with the customer ID embedded therein, to a store employee. When the store employee reads the barcode or QR code with a cash register terminal, the information processing device 300 adds the attribute information linked to the customer ID to the sales information of the product purchased by the customer and stores it in the attribute-specific product purchase information data 510.

[0044] Another example of a method for acquiring customer attribute information is a method for estimating customer attributes by image analysis of an image acquired by the image acquisition unit 321. For example, if there are multiple imaging units 101 in FIG. 5 in the store 100, some of them are installed around the cash register, and the customer attribute estimation unit 501 performs image analysis of an image captured around the cash register to estimate the attributes of a customer who is making a payment. The customer attribute estimation unit 501 of the information processing device 300 may estimate the customer attributes, or a computer provided separately from the information processing device 300 may estimate the customer attributes. In the latter case, the computer may perform image analysis of the acquired image in the same manner as the customer attribute estimation unit 501 in the information processing device 300, and add the customer attribute estimation data obtained to the sales information of the product purchased by the customer. The computer may then store the customer attribute estimation data and the product sales information as a set in the attribute-specific product purchase information data 510. According to this method, even if the customer performs the payment operation at the cash register terminal by himself, the store side can grasp the customer attributes.

[0045] In addition, as another example of a method of acquiring customer attributes, in addition to the method of estimating customer attributes using images captured around the cash register, a method of accumulating data in attribute-specific product purchase information data 510 during normal system operation is also considered. Specifically, the customer attribute estimation unit 501 estimates customer attributes by performing image analysis on images captured around the product shelves. In addition, the behavior determination unit 323 and the product estimation unit 326 estimate which product the customer picked up and put into a shopping basket or cart that the customer has. If it is estimated that the product was put into a shopping basket or cart, the customer attribute estimation unit 501 may add the attribute estimation data to the estimated sales data of the estimated product and store it. With this method, even in a store such as a cashierless store, it is possible to accumulate a set of product sales estimation data and customer attribute estimation data in the attribute-specific product purchase information data 510.

[0046] The attribute-specific product purchase analysis unit 502 uses the attribute-specific product purchase information data 510 to analyze product purchase tendencies for each attribute of the customer, and provides the analysis result to the sentence generation unit 327. Although the attribute-specific product purchase analysis unit 502 is described in the information processing device 300 here, it may be included in a computer provided outside the information processing device 300. For example, it may be included in a server in which the attribute-specific product purchase information data 510 is stored. In other words, the attribute-specific product purchase analysis unit 502 may be included in an external device that can communicate with the information processing device 300 and access a storage medium in which the attribute-specific product purchase information data 510 is stored.

[0047] FIG. 6 is a flowchart showing an example of the flow of processing executed by the information processing device 300 according to this embodiment. Steps S601 to S605 are similar to steps S401 to S405 in FIG. In S606, the customer attribute estimation unit 501 estimates attributes such as the age and sex of the customer 105 from the image acquired in S601. In S607, the attribute-specific product purchase analysis unit 502 analyzes the product purchase tendency for the attribute estimated in S606 from the attribute-specific product purchase information data 510. The process of this step is an example of a process of analyzing the product purchase history for each attribute. The process of this step obtains an analysis result of the purchase history for the attributes of the person in the image. In S608, the sentence generation unit 327 generates sentences (text data) to be communicated to the customer 105 by inputting the behavioral information of the customer 105, the estimated product information of the product 110, and the product purchasing tendency analyzed in S607 into the sentence generation model 350. In addition, when the information processing device 300 determines that a product has been added to the shopping cart based on the behavioral information of the customer 105, the information on the product 110 estimated in S605 may be associated with the customer attributes estimated in S606 and stored in the attribute-specific product purchase information data 510. The process of S609 is the same as S407 in Fig. 4. After that, the series of processes in the flowchart ends.

[0048] The information processing device 300 continuously acquires images of the customer 105 and repeatedly executes a series of steps in the flowchart at a predetermined interval. This makes it possible to monitor the movement of the customer 105 and, at the timing when the customer 105 shows an interest in a product, to transmit to the customer 105 by voice purchase promotion information that matches the purchasing tendency of the attributes to which the customer 105 belongs.

[0049] According to the above-described embodiment 2, in addition to the effect of embodiment 1, purchase promotion information that takes into account the product purchasing tendency of each customer attribute can be automatically generated as text. Therefore, it is expected that the customer's purchasing motivation will be further improved and the average customer spending will be further increased.

[0050] <Embodiment 3> In the store system according to the third embodiment, in addition to the behavioral information of the customer and the product information of the product, the individual information of the customer who showed an interest in the product is used to generate a message to be sent to the customer. In the following, the same parts as in the first embodiment will be omitted, and the differences from the first embodiment will be mainly described.

[0051] Fig. 7 shows a functional configuration of an information processing device 300 according to this embodiment. The information processing device 300 in Fig. 7 differs from the functional configuration of the information processing device 300 in Fig. 3 in that a customer identification unit 701 and a customer-specific product purchase analysis unit 702 are added. The customer identification unit 701 performs image analysis on the face area of ​​the customer 105 in the image acquired by the image acquisition unit 321, and identifies the individual customer from the features of the face area of ​​the customer 105, etc.

[0052] Moreover, the information processing device 300 according to this embodiment uses customer-specific product purchase information data 710. The customer-specific product purchase information data 710 is a data group in which identification information of a customer who purchased a product is added to sales information of the product at the store 100 (such as the product name, sale date and time, and sales quantity of the product sold), and is stored in the external storage device 222. A method for acquiring the customer identification information to be added to the product sales information will be described below.

[0053] For example, there is a method of using an application that is pre-installed on a customer's smartphone and that accumulates discount points according to the purchase price when purchasing a product. When installing this application, the customer's own facial image data is registered, and the customer information including the facial image data is linked to the application ID and stored in a server managed by the store. When the customer pays at the cash register, the customer launches the above-mentioned application and shows a store employee the screen of the smartphone that displays a barcode or QR code with the application ID embedded therein. When the store employee reads this barcode or QR code with the cash register terminal, the information processing device 300 adds the customer information linked to the application ID to the sales information of the product purchased by the customer, and accumulates it in the customer-specific product purchase information data 710.

[0054] Another example of a method for acquiring customer identification information is to identify a customer by performing image analysis on an image acquired by the image acquisition unit 321. For example, if the store 100 has a plurality of image capture units 101 in FIG. 7, some of them may be installed around the cash register, and the customer identification unit 701 may perform image analysis on an image captured around the cash register to identify an individual customer from the features of the face area of ​​the customer who is making a payment. The customer identification unit 701 of the information processing device 300 may identify the customer, or a computer provided separately from the information processing device 300 may identify the customer. In the latter case, the computer may perform image analysis on the acquired image in the same manner as the customer identification unit 701 in the information processing device 300, and add the customer identification information obtained to the sales information of the product purchased by the customer. The computer may then store the customer identification information and the product sales information as a set in the customer-specific product purchase information data 710.

[0055] In addition, as another example of a method of acquiring customer identification information, in addition to the method of identifying customers using images captured around the cash register, a method of storing data in the customer-specific product purchase information data 710 during normal system operation is also considered. Specifically, the customer identification unit 701 performs image analysis on images captured around the product shelves to identify individual customers. In addition, the behavior determination unit 323 and the product estimation unit 326 estimate which product the customer picked up and put into a shopping basket or cart that the customer has. If it is estimated that the product was put into a shopping basket or cart, the customer identification unit 701 may add the customer identification information to the estimated sales data of the estimated product and store it. With this method, even in a store such as a cashierless store, it is possible to store the estimated sales data of the product and the customer identification information as a set in the customer-specific product purchase information data 710.

[0056] The customer-specific product purchase analysis unit 702 uses the customer-specific product purchase information data 710 to analyze the product purchase tendency of each customer, and provides the analysis result to the sentence generation unit 327. Although the customer-specific product purchase analysis unit 702 is described in the information processing device 300 here, it may be included in a computer provided outside the information processing device 300. For example, it may be included in a server in which the customer-specific product purchase information data 710 is stored. In other words, the customer-specific product purchase analysis unit 702 may be included in an external device that can communicate with the information processing device 300 and access a storage medium in which the customer-specific product purchase information data 710 is stored.

[0057] FIG. 8 is a flowchart showing an example of the flow of processing executed by the information processing device 300 according to this embodiment. Steps S801 to S805 are similar to steps S401 to S405 in FIG. In S806, the customer identification unit 701 identifies the individual customer 105 from the image acquired in S801. Specifically, the customer identification unit 701 identifies the ID (identification information) of the customer 105 in the image by comparing a feature amount extracted from the face area of ​​the customer 105 in the image acquired in S801 with a feature amount extracted from face image data registered in association with the application ID. In S807, the customer-specific product purchase analysis unit 702 analyzes the product purchasing tendency of the customer identified in S806 from the customer-specific product purchase information data 710. The process of this step is an example of a process of analyzing the product purchase history of each person. The process of this step provides an analysis result of the purchase history of the person in the image. In S808, the sentence generation unit 327 generates sentences (text data) to be communicated to the customer 105 by inputting the behavioral information of the customer 105, the estimated product information of the product 110, and the product purchasing tendency analyzed in S807 into the sentence generation model 350. In addition, when the information processing device 300 determines that a product has been added to the shopping cart based on the behavioral information of the customer 105, it may associate the information of the product 110 estimated in S805 with the identification information of the customer specified in S806 and store the associated information in the customer-specific product purchasing information data 710. The process of S809 is the same as S407 in Fig. 4. After that, the series of processes in the flowchart ends.

[0058] The information processing device 300 continuously acquires images of the customer 105 and repeatedly executes the series of steps in the flowchart at a predetermined interval. This makes it possible to monitor the movement of the customer 105 and transmit to the customer 105 by voice purchase promotion information that matches the purchasing tendency of the customer 105 at the timing when the customer 105 shows an interest in a product.

[0059] According to the above-described embodiment 3, in addition to the effect of embodiment 1, purchase promotion information that takes into account the product purchasing tendency of each customer can be automatically generated as text. Therefore, it is expected that the customer's purchasing motivation will be further improved and the average customer spending will be further increased.

[0060] <Embodiment 4> In the store system according to the fourth embodiment, voice data tailored to the customer's preferences is generated from the generated text. In the following, the same parts as in the third embodiment will not be described, and the differences from the third embodiment will be mainly described.

[0061] Fig. 9 shows a functional configuration of an information processing device 300 according to this embodiment. The information processing device 300 in Fig. 9 differs from the functional configuration of the information processing device 300 in Fig. 3 in that a customer-specific preference voice analysis unit 901 is added. Furthermore, the information processing device 300 according to this embodiment uses customer-specific preference information data 910. The customer-specific preference information data 910 is data that records preference information data of individual customers. A method of acquiring customer preference information will be described below.

[0062] For example, a method is to use an application of the store 100 that is pre-installed on a smartphone owned by a customer. The information processing device 300 acquires a behavior history (activity information) on the smartphone via the application, and acquires customer preference information from the activity information. The customer preference information is, for example, information on the customer's favorite anime or celebrities. The information processing device 300 links the acquired preference information with the customer's ID, and stores it as customer-specific preference information data 910 in a server managed by the store.

[0063] The customer-specific preference voice analysis unit 901 uses the customer-specific preference information data 910 to analyze the tendency of the preference information of the customer 105 identified by the customer identification unit 701, and provides the analysis result to the information transmission instruction unit 328. Although the customer-specific preference voice analysis unit 901 is described in the information processing device 300 here, it may be included in a computer provided outside the information processing device 300. For example, it may be included in a server in which the customer-specific preference information data 910 is stored. In other words, the customer-specific preference voice analysis unit 901 may be included in an external device that can communicate with the information processing device 300 and access a storage medium in which the customer-specific preference information data 910 is stored.

[0064] The information transmission instruction unit 328 synthesizes a voice signal that matches the preferences of the customer 105 from the text information (text data) generated by the text generation unit 327 based on the analysis result of the preference information of the customer 105 by the customer-specific preference voice analysis unit 901, and transmits the synthesized voice signal to the voice output unit 102. The voice output unit 102 then outputs the voice signal received from the information transmission instruction unit 328. Note that the voice output unit 102 may receive the text information (text data) generated by the text generation unit 327 and the analysis result of the preference information, and change the voice for reading out the text information according to the analysis result of the preference information.

[0065] FIG. 10 is a flowchart showing an example of the flow of processing executed by the information processing device 300 according to this embodiment. Steps S1001 to S1008 are similar to steps S801 to S808 in FIG. In S1009, the customer preference voice analysis unit 901 analyzes the tendency of the preference information of the customer identified in S1006 from the customer preference information data 910. The process of this step is an example of a process of analyzing preferences for each person. By the process of this step, an analysis result of the preferences of the person in the image is obtained. In S1010, the information transmission instruction unit 328 synthesizes a voice signal that matches the customer's preferences identified in S1006 from the text generated in S1008 based on the analysis results obtained in S1009. In S1011, the information transmission instruction unit 328 transmits the voice signal generated in S1010 to the voice output unit 102. The voice output unit 102 receives and outputs the voice signal from the information transmission instruction unit 328, whereby information about the target product 110 is transmitted to the customer 105 in the store 100 in a voice preferred by the customer 105. After that, the series of steps in the flowchart ends.

[0066] According to the above-described embodiment 4, in addition to the effect of embodiment 3, it is possible to switch to a voice that matches the preference of each customer. Therefore, it is expected that the purchasing motivation of customers will be improved and the average customer price will be further improved.

[0067] <Other embodiments> Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of the specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features.

[0068] As a modified example of the above-described embodiment, the product estimation unit 326 may estimate the target product by analyzing an image showing the customer 105, instead of estimating the target product based on the position information of the product in the store 100 and the behavior information of the customer 105. Specifically, the product estimation unit 326 may estimate the target product by, for example, detecting the area of ​​the product held by the customer 105 from the image, and comparing the features of the detected product area with the features of the image data of the product included in the product information data 340.

[0069] As a modification of the third and fourth embodiments, the information transmission instruction unit 328 may transmit an audio signal to a smartphone owned by the customer 105 instead of transmitting an audio signal to the audio output unit 102. In this case, the audio is output via an application of the store 100 installed on the smartphone. Specifically, the information transmission instruction unit 328 transmits an audio signal to the smartphone of the identified customer via the application. As a result, audio representing product information is output from earphones connected to the smartphone that received the audio signal.

[0070] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0071] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) An image acquisition means for acquiring an image from an imaging unit that images the inside of the store; A behavior analysis means for analyzing the behavior of a person in the image; a product estimation means for performing estimation processing of a target product based on position information of the product in the store and the analysis result by the behavior analysis means; a text generation means for performing a generation process for generating text information to be communicated to the person in the image based on the product information of the target product and the analysis result by the behavior analysis means; an output control means for controlling the generated text information to be converted into voice data and output; 13. An information processing device comprising: (Configuration 2) The information processing device according to configuration 1, wherein the sentence generation means performs the generation process by inputting product information of the target product and the analysis results by the behavior analysis means into a trained model. (Configuration 3) The image acquisition means continuously acquires the images from the imaging unit, The information processing device according to configuration 1 or 2, characterized in that, when the analysis result by the behavior analysis means indicates that the person in the image has shown interest in a product in the store, the product estimation means performs the estimation process, the sentence generation means performs the generation process, and the output control means performs processing to output voice data. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the output control means controls to change the type of sound to be output based on product information of the target product. (Configuration 5) The image acquisition means continuously acquires the images from the imaging unit, 5. The information processing device according to any one of configurations 1 to 4, wherein the behavior analysis means analyzes a time-series change in posture of a person in the image. (Configuration 6) An attribute estimation means for estimating attributes of a person in the image; A first analysis means for analyzing a purchasing history of a product for each attribute; Further comprising: The information processing device according to any one of configurations 1 to 5, wherein the sentence generation means performs the generation process based on product information of the target product, the analysis result by the behavior analysis means, and the analysis result by the first analysis means regarding the attributes estimated by the attribute estimation means. (Configuration 7) an identification means for identifying a person in the image; A second analysis means for analyzing the product purchase history of each person; Further comprising: The information processing device according to any one of configurations 1 to 6, characterized in that the sentence generation means performs the generation process based on product information of the target product, the analysis results by the behavior analysis means, and the analysis results by the second analysis means regarding the person identified by the identification means. (Configuration 8) an identification means for identifying a person in the image; A third analytical means for analyzing preferences of each person; and Further comprising: The information processing device of any one of configurations 1 to 7, wherein the output control means converts the voice data to match the preferences of the person in the image based on the analysis results by the third analysis means for the person identified by the identification means. (Configuration 9) a first storage means for storing the attribute estimated by the attribute estimation means and the target product estimated by the product estimation means in association with each other based on the analysis result by the behavior analysis means, 7. The information processing device according to configuration 6, wherein the first analysis means uses the information stored in the first storage means to analyze a purchasing history of a product for each attribute. (Configuration 10) a second storage means for storing, in association with each other, identification information of the person identified by the identification means and the target product estimated by the product estimation means, based on an analysis result by the behavior analysis means; 8. The information processing apparatus according to configuration 7, wherein the second analysis means uses the information stored in the second storage means to analyze a product purchase history for each person. (method) an image acquisition step of acquiring an image from an imaging unit that images the inside of the store; a behavior analysis step of analyzing the behavior of a person in the image; a product estimation step of performing estimation processing of a target product based on position information of the product in the store and the analysis result in the behavior analysis step; a text generation step of performing a generation process of generating text information to be communicated to the person in the image based on product information of the target product and the analysis result in the behavior analysis step; an output control step of controlling the generated text information to be converted into voice data and output; 13. An information processing method comprising: (program) Computer, An image acquisition means for acquiring an image from an imaging unit that images the inside of the store; A behavior analysis means for analyzing the behavior of a person in the image; a product estimation means for performing estimation processing of a target product based on position information of the product in the store and the analysis result by the behavior analysis means; a text generation means for performing a generation process for generating text information to be communicated to the person in the image based on the product information of the target product and the analysis result by the behavior analysis means; an output control means for controlling the conversion of the generated text information into voice data and outputting the voice data; A program characterized by causing the program to function as a [Explanation of symbols]

[0072] 100: store, 101: imaging unit, 102: audio output unit, 103, 104: product shelves, 105: customer, 110: product, 222: external storage device, 351: display unit

Claims

1. An image acquisition means for acquiring an image from an imaging unit that images the inside of the store; A behavior analysis means for analyzing the behavior of a person in the image; a product estimation means for performing estimation processing of a target product based on position information of the product in the store and the analysis result by the behavior analysis means; a text generation means for performing a generation process for generating text information to be communicated to the person in the image based on the product information of the target product and the analysis result by the behavior analysis means; an output control means for controlling the generated text information to be converted into voice data and output; 13. An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the sentence generation means performs the generation process by inputting product information of the target product and the analysis results by the behavior analysis means into a learned model.

3. The image acquisition means continuously acquires the images from the imaging unit, The information processing device according to claim 1, characterized in that, when the analysis results by the behavioral analysis means indicate that the person in the image has shown interest in a product in the store, the product estimation means performs the estimation process, the sentence generation means performs the generation process, and the output control means performs processing to output voice data.

4. 2. The information processing apparatus according to claim 1, wherein the output control means controls so as to change the type of sound to be output based on product information of the target product.

5. The image acquisition means continuously acquires the images from the imaging unit, 2. The information processing apparatus according to claim 1, wherein said behavior analysis means analyzes time-series changes in posture of a person in said image.

6. An attribute estimation means for estimating attributes of a person in the image; A first analysis means for analyzing a purchase history of a product for each attribute; and The information processing device according to claim 1, characterized in that the sentence generation means performs the generation process based on product information of the target product, the analysis result by the behavior analysis means, and the analysis result by the first analysis means regarding the attributes estimated by the attribute estimation means.

7. an identification means for identifying a person in the image; A second analysis means for analyzing the product purchase history of each person; Further comprising: The information processing device according to claim 1, characterized in that the sentence generation means performs the generation process based on product information of the target product, the analysis results by the behavior analysis means, and the analysis results by the second analysis means regarding the person identified by the identification means.

8. an identification means for identifying a person in the image; A third analysis means for analyzing preferences of each person; Further comprising: The information processing device according to claim 1, characterized in that the output control means converts the voice data to match the preferences of the person in the image based on the analysis results by the third analysis means for the person identified by the identification means.

9. a first storage means for storing the attribute estimated by the attribute estimation means and the target product estimated by the product estimation means in association with each other based on an analysis result by the behavior analysis means, 7. The information processing apparatus according to claim 6, wherein the first analysis means analyzes a purchasing history of a product for each attribute by using the information stored in the first storage means.

10. a second storage means for storing, in association with each other, identification information of the person identified by the identification means and the target product estimated by the product estimation means, based on an analysis result by the behavior analysis means; 8. The information processing apparatus according to claim 7, wherein the second analysis means analyzes a product purchase history for each person by using the information stored in the second storage means.

11. an image acquisition step of acquiring an image from an imaging unit that images the inside of the store; a behavior analysis step of analyzing the behavior of a person in the image; a product estimation step of performing estimation processing of a target product based on position information of the product in the store and the analysis result in the behavior analysis step; a text generation step of performing a generation process of generating text information to be communicated to the person in the image based on product information of the target product and the analysis result in the behavior analysis step; an output control step of controlling the generated text information to be converted into voice data and output; 13. An information processing method comprising:

12. Computer, An image acquisition means for acquiring an image from an imaging unit that images the inside of the store; A behavior analysis means for analyzing the behavior of a person in the image; a product estimation means for performing estimation processing of a target product based on position information of the product in the store and the analysis result by the behavior analysis means; a text generation means for performing a generation process for generating text information to be communicated to the person in the image based on the product information of the target product and the analysis result by the behavior analysis means; an output control means for controlling the conversion of the generated text information into voice data and outputting the voice data; A program characterized by causing the program to function as a

Images