Program, method, information processing device, and system

Abstract

To provide a shopping experience with great interest by utilizing AR.SOLUTION: A program is to be executed by a computer comprising a processor and a memory. The program causes the processor to execute the steps of: acquiring data obtained by photographing a carrier that is to carry commodities and that includes, in at least part thereof, an area for generating an object image by augmented reality; creating an object image corresponding to the area included in the acquired data; and displaying the created object image at a position corresponding to an area in an image based on the photographed data.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] In recent years, technologies related to AR (Augmented Reality) and MR (Mixed Reality) have been developed. AR and MR extend the real world seen by the user by adding information to the real space seen by the user. For example, Patent Document 1 proposes a shopping mechanism that allows a user to visually view and order a product even if the product is not actually in front of the user's eyes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-61453 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes the use of AR and MR to make shopping more efficient. However, the use of AR and MR is not limited to making shopping more efficient.

[0005] The purpose of this disclosure is to utilize AR and MR to provide a highly interesting shopping experience. [Means for solving the problem]

[0006] A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the steps of acquiring data obtained by photographing a carrier for transporting goods, the carrier including, at least in part, an area for generating an object image by augmented reality, creating an object image corresponding to the area included in the acquired data, and displaying the created object image at a position corresponding to the area in an image based on the photographed data. Effect of the Invention

[0007] According to the present disclosure, AR can be utilized to provide a highly interesting shopping experience. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Diagram 2] 2 is a block diagram illustrating an example of the configuration of a terminal device 10 shown in FIG. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a server 20. [Figure 4] FIG. 2 is a diagram showing the data structure of a visitor information table 2021. [Diagram 5] 2 shows the data structure of an object information table 2022. FIG. [Figure 6] 2 shows the data structure of a position information table 2023. FIG. [Figure 7] 13 is a diagram showing the data structure of an environment information table 2024. FIG. [Figure 8] FIG. 23 is a diagram showing the data structure of a product information table 2025. [Figure 9] 1 is a schematic diagram illustrating an example of a vehicle 50 on which a terminal device 10 is mounted. [Figure 10] 2 is a diagram for explaining an example of the operation of the terminal device 10 and the server 20 when an object image is displayed on the terminal device 10. FIG. [Figure 11] 2 is a schematic diagram showing an example of an image displayed on a display 141. FIG. [Figure 12] 2 is a schematic diagram showing an example of an image displayed on a display 141. FIG. [Figure 13] 11 is a flowchart showing an example of an operation performed by the terminal device 10 when generating an effect on an object image to be displayed. [Figure 14] 2 is a schematic diagram showing an example of an image displayed on a display 141. FIG. [Figure 15] 2 is a schematic diagram showing an example of an image displayed on a display 141. FIG. [Figure 16] 2 is a schematic diagram showing an example of an image displayed on a display 141. FIG. [Figure 17] 10 is a flowchart showing an example of an operation performed by the server 20 when generating an object image in an image to be displayed on the display 30. [Figure 18] 1A and 1B are schematic diagrams illustrating examples of images based on acquired photographic data. [Figure 19] 2 is a schematic diagram showing an example of an image displayed on a display 30. FIG. [Figure 20] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0010] <Summary> The system according to the present embodiment is installed, for example, in a store where customers can shop. The system recognizes a specific area on a carrier, such as a basket or cart, in which the customer carries merchandise, and generates an object image so that the customer can understand that an object exists in the recognized area. This allows the customer to feel as if they are shopping together with the object, and allows for a fun shopping experience.

[0011] <1 Overall system configuration> Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, a display 30, a three-dimensional measuring device 41, an imaging device 42, and a carrier 50. The terminal device 10, the server 20, the display 30, the three-dimensional measuring device 41, and the imaging device 42 are communicatively connected via, for example, a network 80.

[0012] 1 shows an example in which the system 1 includes one terminal device 10, the number of terminal devices 10 included in the system 1 is not limited to one. The number of terminal devices 10 included in the system 1 may be one or more.

[0013] 1 shows an example in which the system 1 includes one server 20, but the number of servers 20 included in the system 1 is not limited to one. The server 20 may be composed of multiple servers depending on the functions it has. Also, the server 20 may be, for example, a collection of multiple devices that constitutes one server. The method of allocating multiple functions required to realize the server 20 according to this embodiment to one or multiple pieces of hardware can be appropriately determined in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0014] 1 shows an example in which the system 1 includes one display 30, the number of displays 30 included in the system 1 is not limited to one. The number of displays 30 included in the system 1 may be one or more.

[0015] 1 shows an example in which the system 1 includes one each of the three-dimensional measuring device 41 and the imaging device 42, but the number of the three-dimensional measuring device 41 and the imaging device 42 included in the system 1 is not limited to one each. The number of the three-dimensional measuring device 41 and the imaging device 42 included in the system 1 may be one or more.

[0016] 1 illustrates an example in which the system 1 includes one transportation device 50, but the number of transportation devices 50 included in the system 1 is not limited to one. The number of transportation devices 50 included in the system 1 may be one or more.

[0017] The terminal device 10 shown in Fig. 1 is, for example, an information processing device operated by a customer who visits a store. The terminal device 10 is realized by, for example, a mobile terminal such as a smartphone or a tablet. The terminal device 10 may be a wearable terminal such as an HMD (Head Mount Display). The terminal device 10 may be owned by the customer or may be loaned by the store when the customer enters the store.

[0018] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device (e.g., a touch panel, a touch pad, etc.) for receiving an input operation from a user. The output device 14 is a device (a display, a speaker, etc.) for presenting information to a user.

[0019] The server 20 is, for example, an information processing device that manages information about objects to be generated for customers who visit the store.

[0020] The server 20 is realized by, for example, a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an interface with an input device for receiving an input operation from a user and an output device for presenting information to the user.

[0021] The display 30 displays data according to the control of the server 20. The display 30 is installed, for example, in a store and displays images captured inside the store. The display 30 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0022] The three-dimensional measuring device 41 measures the three-dimensional position of an object in the store. For example, the three-dimensional measuring device 41 irradiates a laser onto a predetermined scanning area at a predetermined period and calculates the three-dimensional coordinates of a reflection point by analyzing the reflected light. The three-dimensional measuring device 41 is realized, for example, by LiDAR (Light Detection And Ranging). For example, the three-dimensional measuring device 41 is installed in a position where the entire store can be viewed without obstructions. The three-dimensional measuring device 41 transmits data related to the calculated three-dimensional coordinates to the server 20 as point cloud data.

[0023] The photographing device 42 is a device for receiving light with a light receiving element and outputting the light as photographing data. The photographing device 42 is installed so as to be able to photograph a predetermined position in the store with a predetermined angle of view, for example. The photographing device 42 outputs the photographing data to the server 20.

[0024] The carrier 50 is equipment that allows a customer to transport products within a store. In other words, the carrier 50 is equipment that allows a customer to transport a selected product to a cash register counter. The carrier 50 is, for example, a basket, a cart for placing a basket, or both.

[0025] At least a portion of the vehicle 50 includes an area for generating an object image using augmented reality. The area for generating an object image using augmented reality is, for example, as follows. · The whole or part of the basket (shape and color of the basket) · All or part of the cart (shape of the cart and color of the cart) - Area of ​​the cage with a designated marker Areas of the cart marked with a specific marker

[0026] The marker may be a code in which identification information is written. For example, the marker may be realized by a barcode in which identification information is written, a QR code (registered trademark), or the like. Furthermore, the information written in the marker may be address information for reading information about the object from the server 20. The marker is, for example, affixed to a position where an image can be captured by a camera. Specifically, for example, the marker is affixed to the bottom of the carrier 50 or to the outer side surface of the carrier 50. The marker is not limited to being affixed in one place, and may be affixed in multiple places.

[0027] The fact that at least a portion of the vehicle 50 includes an area for generating an augmented reality object image can be rephrased as, for example, that at least a portion of the vehicle 50 is an identifier for generating an augmented reality object image.

[0028] The carrier 50 may be provided with an arm for supporting the terminal device 10. The angle of the arm is adjusted so that the camera of the supported terminal device 10 faces a direction in which the camera can capture an image of an area for generating an object image by augmented reality. The position at which the arm is provided may be anywhere as long as the camera can capture an image of an area for generating an object image by augmented reality.

[0029] Each information processing device is configured by a computer equipped with a calculation device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0030] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example of the configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected to each other, for example, by a bus or the like.

[0031] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on a signal generated by the control unit 190 and transmits the signal to the outside (for example, the server 20). The communication unit 120 performs reception processing on a signal received from the outside and outputs the signal to the control unit 190.

[0032] The input device 13 is a device for inputting instructions or information by a user who operates the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like in which an instruction is input by touching an operation surface. In the case where the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, a keyboard, a mouse, or the like. The input device 13 converts an instruction input by a user into an electrical signal, and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a receiving port that receives an electrical signal input from an external input device.

[0033] The output device 14 is a device for presenting information to a user who operates the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0034] The audio processing unit 17 performs, for example, digital-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0035] The camera 160 is a device for receiving light with a light receiving element and outputting the received light as an image capturing signal.

[0036] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0037] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181, object information 182, and environmental information 183.

[0038] The user information 181 includes, for example, information about a user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, telephone number, email address, date of registration, store visit information, purchase information, favorite characters, etc. The date of registration indicates, for example, the date on which the user was registered. The store visit information includes a history of the user's visits to a store. The purchase information includes a purchase history of products purchased by the user at a store.

[0039] When the terminal device 10 is lent by a store, the user information 181 may be information capable of identifying the user, such as a user ID, input by a customer as a user. In this case, the user's name, age, address, date of birth, telephone number, email address, date of registration, store visit information, purchase information, favorite character, etc. are managed by the server 20, for example, using the user ID as a key. The server 20 acquires information about the user who has visited the store based on the user ID input from the terminal device 10.

[0040] The object information 182 includes information about an object to be generated in accordance with the transportation device 50. The object information 182 may be stored in advance in the storage unit 180, or may be transmitted from the server 20 based on the photograph of the transportation device 50.

[0041] The environmental information 183 includes information that indicates the environment at the time of the visit. The information that indicates the environment at the time of the visit includes, for example, information about the weather, atmospheric pressure, temperature, humidity, and the like.

[0042] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 operates according to the program to fulfill the functions of an operation reception unit 191, a transmission / reception unit 192, an imaging control unit 193, an image analysis unit 194, and a display control unit 195.

[0043] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131 or the like.

[0044] Furthermore, operation acceptance unit 191 accepts an image input from camera 160. Specifically, operation acceptance unit 191 receives image data captured by camera 160, for example.

[0045] Furthermore, the operation reception unit 191 receives audio information input from the microphone 171. Specifically, for example, the operation reception unit 191 receives audio data that is input from the microphone 171 and converted into digital data by the audio processing unit 17.

[0046] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits instructions input by a user or various pieces of acquired information to the server 20. In addition, the transmitting / receiving unit 192 receives information provided by the server 20.

[0047] The photographing control unit 193 controls photographing by the camera 160. For example, the photographing control unit 193 starts photographing by the camera 160 in response to an instruction from a user.

[0048] The image analysis unit 194 controls the analysis of the photographed data captured by the camera 160. For example, the image analysis unit 194 detects a predetermined area included in the photographed data. Specifically, for example, the image analysis unit 194 detects an area included in the photographed data for generating an object image by augmented reality. Also, for example, the image analysis unit 194 detects a product selected by a user and placed in the carrier 50, which is included in the photographed data. Any existing technology may be used as a method for detection. The image analysis unit 194 may detect an area, a product, etc., using a trained model.

[0049] The display control unit 195 controls the output device 14 to display a predetermined image to the user. For example, the display control unit 195 generates an object image based on information detected from the shooting data, and displays it on the display 141. Specifically, for example, the display control unit 195 generates an object image corresponding to the transportation device 50 being photographed, and displays the object image at a position corresponding to the transportation device 50. At this time, the display control unit 195 superimposes the created object image on a predetermined area of ​​the transportation device 50, for example.

[0050] Furthermore, the display control unit 195 generates a representation according to predetermined information. The predetermined information is, for example, the following. -Products detected in the shooting data · Products you plan to purchase -Past product purchase history -Specified dates (birthdays, anniversaries) Number of past visits Favorite Character ·weather User location information

[0051] <1.2 Functional configuration of the server> 3 is a diagram showing an example of a functional configuration of the server 20. As shown in FIG. 3, the server 20 fulfills the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0052] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0053] The memory unit 202 includes, for example, a visitor information table 2021, an object information table 2022, a position information table 2023, an environmental information table 2024, a product information table 2025, and a trained model 2026.

[0054] The customer information table 2021 is a table that stores information about customers who visit the store. Details will be described later.

[0055] The object information table 2022 is a table for storing information about objects, the details of which will be described later.

[0056] The location information table 2023 is a table for storing the current location of the customer, which will be described in detail later.

[0057] The environment information table 2024 is a table for storing information related to the environment, the details of which will be described later.

[0058] The product information table 2025 is a table that stores information about products.

[0059] The trained model 2026 is, for example, a model for detecting a predetermined area from a captured image. The trained model 2026 is stored in advance when the server 20 provides a service. The trained model 2026 is generated by making a machine learning model perform machine learning according to a model learning program. The trained model 2026 is, for example, a parameterized composite function in which a plurality of functions are combined. The parameterized composite function is defined by a combination of a plurality of adjustable functions and parameters. The trained model 2026 according to this embodiment may be any parameterized composite function that satisfies the above requirements.

[0060] For example, when the trained model 2026 is generated using a forward propagation type multi-layer network, the parameterized composite function is defined as a combination of, for example, a linear relationship between each layer using a weight matrix, a nonlinear relationship (or a linear relationship) using an activation function in each layer, and a bias. The weight matrix and the bias are called parameters of the multi-layer network. The parameterized composite function changes its form as a function depending on how the parameters are selected. In a multi-layer network, by appropriately setting the constituent parameters, it is possible to define a function that can output a desirable result from the output layer.

[0061] As the multi-layered network according to the present embodiment, for example, a deep neural network (DNN) that is a multi-layered neural network that is the subject of deep learning may be used. As the DNN, for example, a convolution neural network (CNN) that targets images may be used.

[0062] The trained model 2026 is trained, for example, with an image of a specific product as input data and with the product name of the product as correct output data. The trained model 2026 is trained, for example, with an image of a basket, cart, or code as input data and with the recognition of "basket," "cart," or "code" for the input image as correct output data. The trained model 2026 may be retrained at any time based on newly added products, etc.

[0063] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The control unit 203 operates according to the program to fulfill the functions of a reception control module 2031, a transmission control module 2032, an image analysis module 2033, a decision module 2034, and a display control module 2035.

[0064] The reception control module 2031 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0065] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0066] The image analysis module 2033 controls the analysis of the photographed data photographed by the photographing device 42. For example, the image analysis module 2033 detects a predetermined area included in the photographed data. Specifically, for example, the image analysis module 2033 detects an area included in the photographed data for generating an object image by augmented reality. Any existing technology may be used as a method for detection. The image analysis module 2033 may detect an area, etc., by using a trained model.

[0067] The determination module 2034 controls the process of determining an object to be generated for each customer. For example, the determination module 2034 determines an object to be generated for a user who owns the terminal device 10 based on information transmitted from the terminal device 10. In addition, the determination module 2034 determines an object to be generated for the carrier 50 included in the captured data based on an analysis result of the captured data captured by the camera device 42.

[0068] The display control module 2035 controls the display 30 to display a predetermined image to the customer. For example, the display control module 2035 generates an object image based on information detected from the shooting data, and displays it on the display 30. Specifically, for example, the display control module 2035 generates an object image corresponding to the transportation device 50 being photographed, and displays the object image at a position corresponding to the transportation device 50. At this time, the display control module 2035 superimposes the created object image on a predetermined area of ​​the transportation device 50, for example.

[0069] In addition, the display control module 2035 generates a presentation according to predetermined information. The predetermined information is, for example, the following. -Past product purchase history -Specified dates (birthdays, anniversaries) Number of past visits Favorite Character ·weather

[0070] <2 Data Structure> 4 to 8 are diagrams showing the data structures of tables stored in the server 20. Note that Fig. 4 to 8 are merely examples and do not exclude data that is not listed. In addition, even data that is listed in the same table may be stored in separate storage areas in the storage unit 202.

[0071] 4 is a diagram showing the data structure of the visitor information table 2021. The visitor information table 2021 shown in FIG. 4 is a table having columns such as name, age, address, date of birth, e-mail address, date of registration, store visit information, purchase information, favorite character, generated object, etc., with the user ID as a key. The visitor information table 2021 may have a column for storing products that the customer plans to purchase.

[0072] The user ID is an item that stores an identifier for uniquely identifying a user. The user ID may be an ID that was set when the user became a member of a store. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The address is an item that stores the place where the user lives. The date of birth is an item that stores the date the user was born. The email address is an item that stores the email address of the user. The registration date is an item that stores the date the user registered as a member.

[0073] The store visit information is an item that stores the history of the user's visits to the store. The store visit information also includes information about the date and time of the current visit. If there are multiple stores, the store visit information may store the date and time of the visit in association with the name of the store visited. The purchase information is an item that stores the history of products purchased by the user. The purchase information includes, for example, the purchase date and time, the product ID of the purchased product, the product name of the purchased product, etc. The favorite character is an item that stores the object ID of a character registered as a favorite by the user. The generated object is an item that stores the object ID of an object set for the user at the time of the current visit. The object is set by the server 20 according to the carrier 50 used by the user.

[0074] The user ID, name, age, address, date of birth, e-mail address, date of registration, visit information, purchase information, and favorite character are stored, for example, in a customer information table (not shown) that is different from the visitor information table 2021. The server 20 extracts information from the customer information table based on the user ID transmitted from the terminal device 10, and stores the extracted information in the visitor information table 2021.

[0075] Fig. 5 is a diagram showing the data structure of the object information table 2022. The object information table 2022 shown in Fig. 5 is a table having columns such as an object name, generation condition, object data, performance content, and requirements corresponding to the performance content, with an object ID as a key.

[0076] The object ID is an item that stores an identifier for uniquely identifying an object. The object name is an item that stores the name of an object. The generation condition is an item that stores the condition for generating an object image. The condition for generating an object image includes, for example, a characteristic aspect of the carrier 50. Specifically, the condition for generating an object image includes, for example, the color of the carrier 50, or identification information written in a code.

[0077] Object data is an item that stores image data about an object. Object data may store reference information (path) to an object data file located in another location. Performance content is an item that stores image data about a performance performed by an object. Performance content may store reference information (path) to an image data file located in another location. Performance content includes, for example, the following: -Performances where objects perform specific actions (birthdays, celebrations of specific dates, etc.) - Guiding (guiding) customers to the desired products based on customer location information and product display information -Evolving objects - The object will wear a designated accessory. -Performance that becomes a fixed object

[0078] In the example shown in FIG. 5, the performance contents include a first performance, a second performance, etc. The first performance may be set to be displayed with priority over the second performance. The requirements are items that store requirements for making the object perform the performance. The requirements include, for example, the following: - You have previously purchased the specified product - You have previously purchased the specified type of product - The number of purchases of the product has exceeded the specified number of times. - The purchase amount of the product exceeds a certain amount. -The current date must match the user's birthday. The current date must match a specific date (e.g. Christmas, Marine Day, New Year's Day, Coming of Age Day, etc.) The current day is a specific day (e.g., Friday). - The number of past visits has exceeded the specified number. The set object (character) matches your favorite character - The weather is as specified (e.g., snow).

[0079] The example shown in FIG. 5 includes a first requirement for implementing a first performance, a second requirement for implementing a second performance, and so on.

[0080] Fig. 6 is a diagram showing the data structure of the position information table 2023. The position information table 2023 shown in Fig. 6 is a table having columns for date and time, user ID, and position information.

[0081] The date and time is an item for storing the date and time when the three-dimensional position of the user was measured by the three-dimensional measuring device 41. The user ID is an item for storing the ID of the user whose three-dimensional position was measured. The position information is an item for storing the measured three-dimensional position.

[0082] Fig. 7 is a diagram showing the data structure of the environment information table 2024. The environment information table 2024 shown in Fig. 7 is a table having columns for date, weather, atmospheric pressure, temperature, and humidity. The environment information table 2024 may include information related to the calendar. The information related to the calendar includes, for example, information related to days of the week and information related to holidays.

[0083] The date is an item for storing the date on which the environmental information was acquired. The weather, atmospheric pressure, temperature, and humidity are examples of the environmental information, and the information provided by a predetermined information source is stored.

[0084] Fig. 8 is a diagram showing the data structure of the product information table 2025. The product information table 2025 shown in Fig. 8 is a table having columns such as product name, price, and display position, with the product ID as a key.

[0085] The item ID is an item that stores an identifier for uniquely identifying a product. The item name is an item that stores the name of the product. The price is an item that stores the price of the product. The display position is an item that stores the position where the product is displayed in the store.

[0086] <3 operations> The operation of the terminal device 10 and the server 20 when providing a customer with an enjoyable shopping experience by displaying object images to the customer will be described.

[0087] In the following description, it is assumed that the carrier 50 is a basket and a cart for placing the basket. In this case, for example, the terminal device 10 is realized by a tablet terminal loaned from a store. The terminal device 10 is fixed so that when a customer holds the handle of the cart to transport a commodity, the display 141 faces the customer and the camera 160 faces in a direction capable of capturing an image of an area for generating an object image by augmented reality.

[0088] 9 is a schematic diagram showing an example of a vehicle 50 equipped with a terminal device 10. The terminal device 10 is fixed so that the display 141 faces the customer and the camera 160 faces in a direction that allows the camera 160 to capture an image of the car.

[0089] (Display process of object image by terminal device 10) FIG. 10 is a diagram for explaining an example of the operations of the terminal device 10 and the server 20 when the terminal device 10 is caused to display an object image.

[0090] In step S11, for example, the terminal device 10 acquires information about a customer who has visited the store. Specifically, for example, the terminal device 10 checks whether the customer has registered as a user, and if registered, requests the customer to input information that can identify the user information, for example, a user ID. If the customer has not registered as a user, the terminal device 10 requests the customer to input information required for user registration. The control unit 190 of the terminal device 10 transmits the information input by the customer to the server 20 via the transmission / reception unit 192. The control unit 190 may transmit the information via a preset application, or may transmit the information using a web browser.

[0091] It should be noted that the customer does not have to use the user information. For example, when the customer is requested by the terminal device 10 to input a user ID or user information, the customer may refuse to input the user ID or user information.

[0092] In step S21, the server 20 registers information about a customer who has visited the store. Specifically, the control unit 203 of the server 20 acquires information about the customer who has visited the store through the reception control module 2031. For example, when the control unit 203 acquires the user ID of the customer, the control unit 203 searches for pre-set user information using the acquired user ID as a key. The control unit 203 registers the user information acquired by the search in the visitor information table 2021. In addition, when user information is input by the customer via the terminal device 10, the control unit 203 registers the input user information in the visitor information table 2021. When the control unit 203 registers information about the customer who has visited the store in the visitor information table 2021, it notifies the terminal device 10 of the registration.

[0093] If the customer refuses to input a user ID or user information, the control unit 203 issues a predetermined ID that is not linked to the user information, and registers the issued ID in the customer information table 2021 as the customer's ID.

[0094] The location in the store where a customer who has come to the store inputs information to start the service may be set in advance. For example, the customer inputs a user ID, user information, or a notice of visiting the store to the terminal device 10 at this reference location. When the server 20 creates a record in the visitor information table 2021 based on the information input by the customer, the server 20 stores the location information at the reference location acquired by the three-dimensional measuring device 41 in the location information table 2023 as the current location information of the customer. In other words, the server 20 links the user information with the location information. The server 20 updates the location of the customer from time to time based on the information from the three-dimensional measuring device 41. This allows the server 20 to grasp the location of the customer in the store. Note that, when acquiring the location information of the customer in the store, permission may be obtained in advance from the customer.

[0095] In step S12, when information about the visiting customer is registered in the visitor information table 2021, the control unit 190 of the terminal device 10 causes the imaging control unit 193 to start imaging with the camera 160. The camera 160 images the carrier 50 at a fixed angle of view. The control unit 190 causes the display control unit 195 to display an image based on the imaging data on the display 141.

[0096] Fig. 11 is a schematic diagram showing an example of an image displayed on display 141. In Fig. 11, camera 160 captures an image of a cart and a basket placed on the cart. As a result, the cart and the basket placed on the cart are displayed on display 141. In the example shown in Fig. 11, code 52 is attached to bottom 51 of the basket. Code 52 is an example of an area for generating an object image by augmented reality.

[0097] In step S22, the control unit 203 acquires information about the transportation device 50 used by the customer.

[0098] Specifically, for example, the control unit 203 acquires image data captured by the terminal device 10 through the reception control module 2031. The control unit 203 analyzes the captured image through the image analysis module 2033. For example, the image analysis module 2033 detects a predetermined area included in the captured data. More specifically, for example, the image analysis module 2033 detects the code 52 included in the captured data. The image analysis module 2033 reads information written in the detected code 52.

[0099] Furthermore, for example, the reception control module 2031 may obtain information written in the code 52 from the terminal device 10. Specifically, for example, the control unit 190 of the terminal device 10 analyzes the image captured by the camera 160 using the image analysis unit 194. Specifically, for example, the image analysis unit 194 detects the code 52 included in the captured data. The image analysis unit 194 reads the information written in the detected code 52. The control unit 190 transmits the read information to the server 20 using the transmission / reception unit 192.

[0100] In step S23, the control unit 203 uses the determination module 2034 to determine an object to be generated for the customer. Specifically, for example, the determination module 2034 acquires information described in the code 52. The determination module 2034 searches for an object that satisfies the generation condition by comparing the condition stored in the item "generation condition" of the object information table 2022 with the read information. The determination module 2034 determines the object that satisfies the generation condition as the object to be presented to the customer. When the object to be presented to the customer is determined, the determination module 2034 stores the object ID of the determined object in the item "generation object" of the record of the corresponding customer in the visitor information table 2021.

[0101] In step S24, the control unit 203 transmits information about the object to the terminal device 10 via the transmission control module 2032. The information about the object includes, for example, an object ID, an object name, a generating condition, object data, a performance content, and a requirement corresponding to the performance content. When the information about the object is pre-stored in the terminal device 10, the transmission control module 2032 may transmit, for example, only the object ID of the determined object to the terminal device 10.

[0102] In step S13, the terminal device 10 displays an object image in the captured data. Specifically, when the control unit 190 of the terminal device 10 receives the object information from the server 20, the control unit 190 analyzes the image captured by the camera 160 by the image analysis unit 194. More specifically, for example, the image analysis unit 194 detects a predetermined area included in the captured data, for example, the code 52 included in the captured data. The image analysis unit 194 reads the information described in the detected code 52.

[0103] The control unit 190 reads out object data from the object information 182 based on the read information using the display control unit 195. The display control unit 195 creates an object image based on the read object data. The display control unit 195 displays the created object image on the display 141. For example, the display control unit 195 displays the generated object image at a position corresponding to the code 52. Specifically, for example, the display control unit 195 superimposes the created object image 53 at a position corresponding to the code 52.

[0104] Fig. 12 is a schematic diagram showing an example of an image displayed on the display 141. In Fig. 12, the display control unit 195 causes the display 141 to display an object image 53 so that a character, which is an example of an object, is present in a cage.

[0105] The display control unit 195 generates a predetermined effect for the object image 53.

[0106] (Object image rendering process in terminal device 10) Fig. 13 is a flowchart showing an example of the operation of the terminal device 10 when generating effects on the object image to be displayed. Note that in Fig. 13, steps S14 to S16 explain another example of the process of displaying the object image in step S13 shown in Fig. 10.

[0107] In step S14, the display control unit 195 acquires information related to the presentation. Specifically, the display control unit 195 acquires, using the user ID as a key, the date of birth, visit information, purchase information, favorite character, information related to products to be purchased today, and the like, from the visitor information table 2021. The display control unit 195 also acquires information related to the customer's current location from the position information table 2023. The display control unit 195 also acquires information related to the weather, information related to the calendar, and the like, from the environment information table 2024. The display control unit 195 also acquires information related to the display position of products from the product information table 2025.

[0108] In step S15, the display control unit 195 determines the effect to be performed by the object image. Specifically, the display control unit 195, for example, compares the acquired information on the effect with the requirements for generating the effect stored in the object information 182. The display control unit 195 determines the effect that satisfies the requirements as the effect to be performed by the object image. When there are multiple effects that satisfy the requirements, the display control unit 195 may determine, for example, the effect that is set to have a high priority as the effect to be performed by the object image.

[0109] In step S16, the display control unit 195 displays an object image reflecting the effect on the shooting data. Specifically, the display control unit 195 reads object data from the object information 182 based on the information read by the image analysis unit 194. The display control unit 195 also reads effect contents for the determined effect from the object information 182. The display control unit 195 creates an object image reflecting the effect based on the read object data and the read effect contents. The display control unit 195 displays the created object image on the display 141. For example, the display control unit 195 displays the generated object image at a position corresponding to the code 52. Specifically, for example, the display control unit 195 superimposes the created object image 53 at a position corresponding to the code 52.

[0110] In step S16, an example has been described in which the display control unit 195 creates an object image reflecting the determined effect and displays the created object image on the display 141, but the present invention is not limited to this. The display control unit 195 may create an object image reflecting the determined effect in response to an operation input from a customer and display the created object image on the display 141. In addition, when there are multiple effects that satisfy the requirements, the display control unit 195 may, for example, accept a selection of an effect from a customer, create an object image reflecting the selected effect, and display the created object image on the display 141.

[0111] For example, the display control unit 195 causes the display 141 to display the following image: - Evolved object images based on store visits or product purchase information -Images of objects wearing designated accessories based on the number of visits to the store or information on product purchases - Object images with an increased number based on information about store visits or product purchases - Object images that guide customers to their desired products based on customer location information and product display information -Images of objects dressed to celebrate a birthday - Object images that perform actions to celebrate a birthday -Images of objects dressed to celebrate a particular day -Images of objects performing actions to celebrate a particular day -Image of an object dressed to indicate a specific day of the week -An object image that performs an action to indicate that it is a specific day of the week A dressed up object image to show that the set object matches your favorite character An object image that moves to show that the set object matches your favorite character -An image of an object dressed to indicate that the weather is a certain weather (e.g., snow) -An object image that moves to indicate that the weather is as specified

[0112] Fig. 14 is a schematic diagram showing an example of an image displayed on the display 141. In Fig. 14, the display control unit 195 causes the display 141 to display an object image 53 such that a character, which is an example of an object, exists in a cage in an evolved state.

[0113] FIG. 15 is a schematic diagram showing an example of an image displayed on the display 141. In FIG. 15, the display control unit 195 displays the object image 53 on the display 141 so that, for example, a character, which is an example of an object, guides the customer to a desired product, so that the character points in the direction of the product. The display control unit 195 receives, for example, information about a product to be purchased from the customer. The control unit 190 acquires information about the customer's current position acquired by the three-dimensional measuring device 41 and the display position of the product that the customer plans to purchase from the server 20. The display control unit 195 creates a route from the customer's current position to the display position of the product. The display control unit 195 displays a guidance display 54 that follows the created route on the display 141. Note that the display for guidance is not limited to the above. A predetermined object related to the object displayed on the carrier 50 may be displayed at the display position of the target product.

[0114] Furthermore, the display control unit 195 is not limited to displaying the object image 53 in the basket. For example, when the terminal device 10 is not fixed to a cart, the display control unit 195 may display the object image 53 on the display 141 so that a character, which is an example of an object, jumps out of the basket and moves to a desired product. This allows, for example, the character to leave the basket and guide the customer to the product.

[0115] In step S17, the image analysis unit 194 analyzes the image captured by the camera 160. Specifically, for example, the image analysis unit 194 detects a product to be placed in the carrier 50 from among the products included in the captured data. The product to be placed in the carrier 50 can be said to be a product placed in a basket. The image analysis unit 194 detects that the product placed in the carrier 50 is a predetermined product, for example, by using a trained model.

[0116] In step S18, the display control unit 195 determines the effect to be performed by the object image. Specifically, the display control unit 195, for example, matches the detected product with the requirements for generating the effect stored in the object information 182. The display control unit 195 determines the effect, which requires the detection of a specific product, as the effect to be performed by the object image. The effect may be set differently for each product.

[0117] In step S19, the display control unit 195 displays an object image reflecting the effect on the shooting data. Specifically, the display control unit 195 reads out the effect content for the determined effect from the object information 182. The display control unit 195 reflects the read out effect content on the displayed object image.

[0118] Fig. 16 is a schematic diagram showing an example of an image displayed on the display 141. In Fig. 16, the display control unit 195 causes the display 141 to display the object image 53 so that the object takes an action corresponding to the product 55 placed in the basket. In Fig. 16, the object image 53 performs an action of being happy when the product 55 is placed in the basket.

[0119] (Display process of object image on display 30 by server 20) FIG. 17 is a flowchart showing an example of the operation of the server 20 when generating an object image in an image to be displayed on the display 30.

[0120] In step S31, the control unit 203 of the server 20 acquires imaging data by the reception control module 2031. Specifically, the reception control module 2031 acquires imaging data captured by the imaging device .

[0121] Fig. 18 is a schematic diagram showing an example of an image based on acquired photographic data. In Fig. 18, the photographing device 42 photographs, for example, a plurality of carts and a basket placed on the cart. As a result, the plurality of carts and the basket placed on the cart are displayed on the display 30. In the example shown in Fig. 18, a code 52 is attached to the side of the basket. The code 52 is an example of an area for generating an object image by augmented reality.

[0122] In step S32, the control unit 203 analyzes the acquired image by the image analysis module 2033. For example, the image analysis module 2033 detects a predetermined area included in the captured data. More specifically, for example, the image analysis module 2033 detects the code 52 included in the captured data. The image analysis module 2033 reads information written in the detected code 52.

[0123] In step S33, the control unit 203 uses the determination module 2034 to determine an object to be generated for the customer. Specifically, for example, the determination module 2034 acquires information described in the code 52. The determination module 2034 searches for an object that satisfies the generation condition by comparing the conditions stored in the item "generation condition" of the object information table 2022 with the read information. The object that satisfies the generation condition is determined as the object to be presented to the customer.

[0124] In step S34, the display control module 2035 acquires information related to the performance. For example, the display control module 2035 identifies the customer to whom the object is assigned based on the determined object. Specifically, the display control module 2035 acquires the user ID of the customer to whom the object is assigned from the visitor information table 2021 using the object ID as a key. The display control module 2035 acquires the date of birth, visit information, purchase information, favorite character, information about products to be purchased today, and the like associated with the acquired user ID from the visitor information table 2021. In addition, the display control module 2035 acquires information related to the customer's current location from the position information table 2023 based on the acquired user ID. In addition, the display control module 2035 acquires information related to the weather, information related to the calendar, and the like from the environment information table 2024.

[0125] In step S35, the display control module 2035 determines the effect to be performed by the object image. Specifically, the display control module 2035, for example, compares the acquired information on the effect with the requirements for generating the effect stored in the object information table 2022. The display control module 2035 determines the effect that satisfies the requirements as the effect to be performed by the object image. When there are multiple effects that satisfy the requirements, the display control module 2035 may, for example, determine the effect that is set to have a high priority as the effect to be performed by the object image.

[0126] In step S36, the display control module 2035 displays an object image reflecting the effect on the shooting data. Specifically, the display control module 2035 reads object data from the object information table 2022 based on the information read by the image analysis module 2033. The display control module 2035 also reads effect contents for the determined effect from the object information table 2022. The display control module 2035 creates an object image reflecting the effect based on the read object data and the read effect contents. The display control module 2035 displays the created object image on the display 30. For example, the display control module 2035 displays the generated object image at a position corresponding to the code 52. Specifically, for example, the display control module 2035 superimposes the created object image 53 at a position corresponding to the code 52.

[0127] Fig. 19 is a schematic diagram showing an example of an image displayed on the display 30. In Fig. 19, the display control module 2035 causes the display 30 to display an object image 53 so that a character, which is an example of an object, is present on the carrier 50 included in the image based on the shooting data.

[0128] In addition, in FIG. 19, the display control module 2035 may display a character, which is an example of an object, not only inside the basket but also outside the basket. Outside the basket indicates, for example, a position outside the basket and in the vicinity of the carrier 50. For example, a customer may leave the carrier 50 at a predetermined position and go to pick up a desired product. The display control module 2035 may move the character out of the basket and toward the customer so as to follow the customer who has left the carrier 50. Furthermore, when the customer uses the carrier 50 again, the display control module 2035 may move the character so as to put it back into the basket. This allows the customer to enjoy shopping with the feeling of carrying the character around with them.

[0129] In the explanation of Fig. 9 to Fig. 16, a tablet terminal provided by a store is used as the terminal device 10. However, the terminal device 10 used in the store may be a smartphone owned by a customer. In this case, the customer can experience the shopping described in Fig. 9 to Fig. 16 by installing a predetermined application on the smartphone or by accessing a predetermined site using a web browser.

[0130] As described above, in the above embodiment, the carrier 50 for carrying goods includes at least a part of an area for generating an object image by augmented reality. The camera 160 or the photographing device 42 photographs an area including the carrier 50. The display control unit 195 or the display control module 2035 creates an object image corresponding to an area included in the photographed area. The display control unit 195 or the display control module 2035 displays the created object image at a position corresponding to an area in the image of the area. This allows an object by augmented reality to be generated in the basket, providing an experience of shopping with the object. In addition, since the object image generated is different for each carrier 50, the next visit to the store will be more enjoyable.

[0131] Therefore, the system according to this embodiment can utilize AR to provide a highly interesting shopping experience.

[0132] In the above embodiment, the display control unit 195 or the display control module 2035 generates a predetermined effect on the object image, which makes shopping more interesting.

[0133] In the above embodiment, the terminal device 10 or the server 20 acquires user information about the customer who visits the store. The display control unit 195 or the display control module 2035 generates effects on the object image based on the acquired user information. This increases the interest of shopping because effects are generated on the object image based on birthdays, store visit history, purchase history, and the like. In addition, because customers can enjoy effects that are different from those of other customers, this can motivate them to visit the store and to purchase products.

[0134] In the above embodiment, the terminal device 10 or the server 20 acquires environmental information. The display control unit 195 or the display control module 2035 generates effects on the object image based on the acquired environmental information. This generates effects on the object image based on information on the environment such as the calendar and weather, which further increases the interest of shopping.

[0135] In the above embodiment, the server 20 acquires user information about customers who visit the store. The three-dimensional measuring device 41 measures the position of the customer in the store. The server 20 associates the user information with the customer's position. This makes it possible to understand how the customer moved around the store, and allows for a deeper consideration of the flow of customers in the store.

[0136] In the above embodiment, the display control unit 195 or the display control module 2035 generates effects on the object image based on the user information and the customer's position. This makes it possible to generate effects according to the customer's position, which further increases the interest of shopping.

[0137] In the above embodiment, the image capturing device 42 captures an image of an area including a plurality of carriers 50. The display control module 2035 creates object images corresponding to a plurality of areas of the plurality of carriers included in the area. The display control module 2035 displays the created object images at positions corresponding to the plurality of areas in the image of the area.

[0138] In the above embodiment, the transmitting / receiving unit 192 acquires photographed data of the product to be placed in the carrier 50. The display control unit 195 generates an effect on the object image based on the product placed in the carrier 50. This makes shopping more enjoyable because the object moves when the product is placed in the basket.

[0139] In the above embodiment, the display control unit 195 varies the effects that are generated in the object images depending on the product. This makes shopping even more fun because the objects behave according to the products that are placed in the basket.

[0140] In the above embodiment, the terminal device 10 acquires user information about a customer who visits the store. The terminal device 10 acquires location information that measures the location of the customer in the store. The display control unit 195 generates an effect on the object image that guides the customer to the product desired by the customer based on the user information and the location information. This makes even the act of searching for the desired product fun.

[0141] <Modification> In the above embodiment, an example has been described in which information is read from a code attached to the carrier 50, and an object image is determined based on the read information. However, the determination of the object image is not limited to being based on the information read from the code. For example, the image analysis unit 194 or the image analysis module 2033 analyzes an image to detect the carrier 50 and acquire features of the detected carrier 50. The determination module 2034 may determine an object image based on the acquired features of the carrier 50. Specifically, for example, the image analysis unit 194 or the image analysis module 2033 analyzes an image and detects a basket from the captured data. The image analysis unit 194 or the image analysis module 2033 acquires the color of the detected basket. The determination module 2034 determines an object image based on the acquired color of the basket.

[0142] In the above embodiment, when the customer finishes shopping, an effect of the object going home may be performed. Specifically, for example, the display control unit 195 or the display control module 2035 causes the display to display an effect of the object going home in the following cases: When the carrier 50 is returned to its designated position When the basket is returned to its designated position When you hand the basket over to the cashier at the counter The effect of an object returning includes, for example, an effect in which a character, which is an example of an object, returns to a predetermined residence, an effect in which a character returns to a predetermined state, etc. By generating an effect in which an object returns, it feels as if you are meeting the same individual object every time you visit the store, making visiting the store more enjoyable.

[0143] Information on the interaction with the object on that day may be stored as a predetermined parameter of the user information, for example. If the object is a character, the predetermined parameter may be represented as the degree of familiarity of the character with the customer, for example.

[0144] Also, in the above embodiment, the influence of objects on each other is not described. However, object images generated based on different regions may react to each other. Specifically, for example, the display control unit 195 or the display control module 2035 recognizes that customers have approached each other based on the position information of the customers. For example, the display control unit 195 or the display control module 2035 recognizes that the customers have passed each other or run side by side. The display control unit 195 or the display control module 2035 controls the object image so that the object takes a predetermined action in the direction in which the customers have approached each other. This makes it look like the objects are interacting with each other, such as by greeting each other, making shopping more enjoyable.

[0145] In addition, when an object image not associated with a basket is displayed, the object images may react to each other. The object image not associated with a basket includes, for example, an object image associated with a display shelf of a product, an object image associated with an entrance, an object image associated with a predetermined monument, and the like. For example, when a customer moves to the vicinity of an area for generating an object image not associated with a basket, the display control unit 195 or the display control module 2035 recognizes that the customer has approached the area based on the customer's position information. The display control unit 195 or the display control module 2035 controls the object image so that the object associated with the basket takes a predetermined action in the direction in which the area is located. In addition, the display control unit 195 or the display control module 2035 controls the object image so that the object not associated with the basket takes a predetermined action in the direction in which the customer is present.

[0146] Also, in the above embodiment, there is no description of an object related to another customer moving toward itself. However, an object related to another customer may move toward itself. Specifically, for example, the display control unit 195 or the display control module 2035 recognizes that customers are close to each other based on the position information of the customers. The display control unit 195 or the display control module 2035 controls the object image so that the object takes a predetermined action toward the other customer based on the user information of the other customer who is close. This makes it possible to make the object take a special action depending on, for example, the circumstances of the other customer, such as the day being a birthday, making shopping more enjoyable.

[0147] Also, in the above embodiment, there is no description of the object image acting in response to the store clerk. However, the object image may react to the store clerk. Specifically, for example, the display control unit 195 or the display control module 2035 recognizes that the customer and the store clerk have come close to each other based on the position information of the store clerk. The display control unit 195 or the display control module 2035 controls the object image so that the object takes a predetermined action in the direction in which the customer and the store clerk have come close to each other. This makes it appear that the object is interacting with the store clerk, such as by greeting the store clerk, making shopping more enjoyable.

[0148] <4 Basic hardware configuration of computer> 20 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main storage device 92, an auxiliary storage device 93, and a communication IF (interface) 99. These are electrically connected to each other by a bus.

[0149] The processor 91 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0150] The main storage device 92 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0151] The auxiliary storage device 93 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0152] The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.

[0153] The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0154] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0155] <Basic functional configuration of computer 90> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration of a computer 90 shown in Fig. 20. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0156] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0157] The control unit is realized by the processor 91 reading out various programs stored in the auxiliary storage device 93, expanding the programs in the main storage device 92, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0158] The storage unit is realized by a main storage device 92 and an auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Furthermore, the processor 91 can secure a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 in accordance with a program. Furthermore, the control unit can cause the processor 91 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0159] A database refers to a relational database, which is used to manage sets of data called tables, which are structured according to rows and columns, by relating them to each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.

[0160] Usually, a column is set in each table as a key for uniquely identifying a record, but setting a key in the column is not essential. The control unit can cause the processor 91 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

[0161] The communication unit is realized by the communication IF 99. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 91 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0162] Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are within the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention.

[0163] <Additional Notes> The matters described in the above embodiments will be supplemented below. (Appendix 1) A system comprising: a vehicle for transporting goods, the vehicle including, at least in part, an area for generating an object image using augmented reality; a means for photographing an area including the vehicle; a means for creating an object image corresponding to the area included in the area; and a means for displaying the created object image at a position corresponding to the area in the image of the area. (Appendix 2) The display means generates a predetermined effect on the object image. (Appendix 3) The system according to claim 2, further comprising a means for acquiring user information about a customer who visits the store, and a display means for generating an effect on the object image based on the acquired user information. (Appendix 4) The system according to claim 2, further comprising: a means for acquiring environmental information; and a display means for generating an effect on the object image based on the acquired environmental information. (Appendix 5) The system described in (Appendix 2) includes a means for acquiring user information about customers who visit a store, a means for measuring the customer's position within the store, and a means for associating the user information with the customer's position. (Appendix 6) The display means generates an effect on the object image based on the user information and the customer's position (a system described in Appendix 5). (Appendix 7) A system described in any one of (Appendix 1) to (Appendix 4), wherein the photographing means photographs an area including a plurality of transportation devices, the creating means creates object images corresponding to a plurality of areas of the plurality of transportation devices included in the area, and the displaying means displays the created object images at positions corresponding to the plurality of areas in the image of the area. (Appendix 8) A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the steps of acquiring data obtained by photographing a transport device for transporting goods, the transport device including, at least in part, an area for generating an object image using augmented reality, creating an object image corresponding to the area included in the acquired data, and displaying the created object image at a position corresponding to the area in an image based on the photographed data. (Appendix 9) The step of acquiring data of an image of the carrier includes acquiring data of an image of a product to be placed in the carrier, and the step of displaying includes generating an effect based on the product in the object image (Appendix 8). (Appendix 10) The display step varies the effect generated in the object image depending on the product (Appendix 9). (Appendix 11) A program described in Appendix 8 that causes a processor to execute a step of acquiring user information about customers who visit a store and a step of acquiring location information that measures the customer's location within the store, and a step of displaying the information includes generating an effect in an object image that guides the customer to the product that the customer desires, based on the user information and the location information. (Appendix 12) A method executed by a computer having a processor and a memory, the method including the steps of: acquiring data obtained by capturing an image of a carrier for transporting goods, the carrier including, at least in part, an area for generating an object image by augmented reality; creating an object image corresponding to the area included in the acquired data; and displaying the created object image at a position corresponding to the area in an image based on the captured data. How to do it. (Appendix 13) An information processing device comprising a control unit and a memory unit, wherein the control unit executes the steps of acquiring data obtained by photographing a carrier for transporting goods, the carrier including, at least in part, an area for generating an object image using augmented reality, creating an object image corresponding to the area included in the acquired data, and displaying the created object image at a position corresponding to the area in an image based on the photographed data. [Explanation of symbols]

[0164] 1. System 10...Terminal device 12…Communication Interface 120…Communications Department 13...Input device 131...Touch-sensitive devices 14...Output device 141…Display 15…Memory 150...Location information sensor 16…Storage 160…Camera 17...Audio processing unit 171…Mike 172…Speaker 180...Storage section 19…Processor 190...Control unit 20…Server

Claims

1. A carrier for carrying an item, the carrier including, at least in a portion thereof, an area for generating an object image by augmented reality; A means for photographing an area including the transportation device; means for generating an object image corresponding to the area included in the range; a means for displaying an area including the photographed transportation device and displaying the created object image inside the transportation device being displayed; A system comprising:

2. The system according to claim 1 , wherein said display means generates a predetermined effect on said object image.

3. A means for acquiring user information about a customer who visits the store is provided, The system according to claim 2 , wherein the display means generates an effect on the object image based on the acquired user information.

4. A means for acquiring environmental information is provided, The system according to claim 2 , wherein the display means generates an effect on the object image based on the acquired environmental information.

5. A means for acquiring user information about a customer who visits the store; A means for determining the location of customers within the store; means for associating the user information with the location of the customer; 3. The system of claim 2, comprising:

6. 6. The system according to claim 5, wherein said display means generates an effect on said object image based on said user information and the position of said customer.

7. A plurality of carriers for carrying goods, the carriers including, at least in part, an area for generating an augmented reality image of an object; A means for photographing an area including a plurality of the vehicles; a means for generating object images corresponding to a plurality of areas of a plurality of the vehicles included in the range; a means for displaying an area including the plurality of photographed conveyances and displaying the created object image inside the plurality of displayed conveyances; A system comprising:

8. A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: acquiring data of an image of a vehicle for transporting goods, the vehicle including, at least in part, an area for generating an object image by using augmented reality; creating an object image corresponding to the region included in the acquired data; displaying an area including the photographed transportation device, and displaying the created object image inside the transportation device being displayed; A program that executes the following.

9. The step of acquiring data of an image of the carrier includes acquiring data of an image of a commodity to be placed in the carrier, 9. The program according to claim 8, wherein the display step generates an effect based on the product in the object image.

10. 10. The program according to claim 9, wherein the display step varies the effect generated on the object image depending on the product.

11. Acquiring user information about a customer who visits the store; acquiring location information measuring the location of customers in the store; causing the processor to execute 9. The program according to claim 8, wherein the display step causes the object image to generate an effect that guides the customer to a product desired by the customer based on the user information and the position information.

12. 1. A computer-implemented method comprising a processor and a memory, the processor comprising: acquiring data of an image of a vehicle for transporting goods, the vehicle including, at least in part, an area for generating an object image by using augmented reality; creating an object image corresponding to the region included in the acquired data; displaying an area including the photographed transportation device, and displaying the created object image inside the transportation device being displayed; How to do it.

13. An information processing device including a control unit and a storage unit, acquiring data of an image of a vehicle for transporting goods, the vehicle including, at least in part, an area for generating an object image by using augmented reality; creating an object image corresponding to the region included in the acquired data; displaying an area including the photographed transportation device, and displaying the created object image inside the transportation device being displayed; An information processing device that executes the above.

Images