System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image of "today's recommended product" by a terminal of a participating store; means for recording a product description and supplementary information of a limited-time sale by voice by the terminal; means for receiving the image and voice data transmitted from the terminal by a server; means for extracting product information by the server using image recognition; means for extracting supplementary information from the voice data by the server using voice recognition; means for aggregating information of all stores acquired by the server and automatically generating a leaflet; and means for electronically distributing the leaflet generated by the server to a pre-registered user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Small stores in shopping districts are at a disadvantage in terms of information dissemination and advertising when competing with large supermarkets. As a result, consumers tend to rely on supermarket flyers, and there is a problem that attractive products and sales information offered by each store in the shopping district is not fully conveyed. This increases the likelihood that consumers will miss out on better deals, and there is a risk that the vitality of the entire shopping district will decrease. [Means for solving the problem]

[0005] This invention provides a means for terminals at participating stores to take images of "Today's Recommended Products" and record audio descriptions of the products and supplemental information about limited-time sales. The server receives the images and audio data sent from the terminals, extracts product information using image recognition, and extracts supplemental information using audio recognition. The server then aggregates the acquired information from all stores, automatically generates flyers, and electronically distributes them to pre-registered users. This series of processes enables each store in the shopping district to provide valuable information to consumers effectively and at low cost, even without large-scale advertising means.

[0006] A "terminal" is an electronic device that allows a store clerk to take pictures of product information and record audio, and is a device that has a communication function for transmitting data to a server.

[0007] The "server" is an information processing device that receives image and audio data sent from the terminal, analyzes this data, generates flyers, and distributes them to users.

[0008] "Product information" is detailed information provided to consumers, such as the name, price, and image of the product sold by the store.

[0009] "Supplementary information" refers to information about additional explanations or conditions that are added to product information, such as time of sale or special promotional content.

[0010] "Image recognition" is a technology that identifies specific objects or features from input image data and extracts related information.

[0011] "Speech recognition" is a technology that analyzes voice data and converts it into text data, and is used to extract information from speech.

[0012] A "flyer" is an advertising medium that is generated based on product information and supplementary information collected from each store in a shopping district and distributed to consumers.

[0013] "Electronic distribution" is the process of sending digitally generated flyers to consumers via the internet, primarily using messaging services such as LINE. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0036] 1. Information collection via devices

[0037] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The captured images clearly display the products, making them attractive and easy to understand for consumers. The terminal also has a voice recording function, allowing the clerk to record a voice explanation of the products and supplementary information about the limited-time sale. The terminal stores this data locally and then transmits it to the server.

[0038] 2. Information analysis by the server

[0039] The server receives the image and audio data sent from the terminal. It uses image recognition technology to extract product information (such as product name and price) from the received image data. The server also uses voice recognition technology to convert the audio data into text, from which it extracts the time of limited-time sales and other important supplementary information. This makes it possible to integrate and consolidate information for the entire shopping district, rather than just for individual stores.

[0040] 3. Generate flyers

[0041] The server automatically generates electronic flyers based on the information collected from each store. These flyers contain product names, prices, and time sale information, and are laid out so that users can easily see all the useful information at a glance. These flyers are output as electronic documents in PDF or image format.

[0042] 4. Electronic Delivery

[0043] The server obtains the LINE account information of pre-registered users. Once the flyer is generated, it is distributed using the LINE API. Users can receive daily updated discount information on their smartphones, tablets, and other devices. This allows stores in shopping districts to deliver the latest information directly to consumers without having to use large-scale advertising methods.

[0044] Specific examples

[0045] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0046] The above is a specific embodiment for carrying out the present invention. The present invention allows consumers to easily obtain more advantageous product information, and each store in the shopping mall can efficiently attract customers.

[0047] The processing flow will be explained below.

[0048] Step 1:

[0049] The device takes images and videos of "Today's Recommended Products." The store clerk takes still and video images of the products shown to the device's camera, which are saved as image files. Next, the store clerk records audio into the device explaining the products and supplementary information about the limited-time sale, which is saved as an audio file.

[0050] Step 2:

[0051] The device sends the captured image and audio files to the server. Specifically, the data is sent using an HTTP POST request. This request includes the store ID, image file, and audio file, and the server temporarily stores them.

[0052] Step 3:

[0053] The server analyzes the image data received from the device and uses image recognition technology to extract product names, prices, and other relevant information, for example by recognizing text in the image and obtaining the necessary information from product labels and price displays.

[0054] Step 4:

[0055] The server analyzes the voice data received from the device. It uses voice recognition technology to convert the voice data into text. From the converted text, it extracts supplemental information such as time of sale and special promotional content. This information is recorded in conjunction with the product information.

[0056] Step 5:

[0057] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. This allows information from multiple stores to be aggregated into a single flyer.

[0058] Step 6:

[0059] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create a visually easy-to-understand flyer. This flyer is output as a PDF or image file.

[0060] Step 7:

[0061] The server obtains the LINE account information of pre-registered users. It then distributes flyers generated using the LINE API to each user. Users can receive the flyers on their smartphones, tablets, or other devices and check the latest daily information.

[0062] The above steps complete a series of processes in which the terminal collects information, the server analyzes and aggregates the information, generates a flyer, and distributes it to the user.

[0063] Example 1

[0064] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0065] Currently, there is a lack of means for each store in a shopping district to efficiently distribute discount information to consumers at low cost. Conventional advertising methods require large amounts of advertising expenses, which places a heavy burden on small stores in shopping districts. In addition, there are limited means of providing consumers with the latest information in a timely manner. This creates challenges for the shopping district as a whole in attracting customers and communicating with consumers.

[0066] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0067] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio for product descriptions and supplemental information about limited-time sales, means for the terminal to transmit image data and audio data collected by the terminal to the server, means for the server to receive the image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate a flyer in electronic document format, and means for electronically distributing the flyer generated by the server to pre-registered users. This enables each store in the shopping district to provide consumers with the latest information effectively and at low cost, thereby increasing its customer attraction.

[0068] A "participating store terminal" is a device with image capture and audio recording functions that can be operated by store staff at each store in the shopping district.

[0069] "Image of today's recommended product" is a photograph or image file that shows the visual information of the product that each store in the shopping district wants to particularly promote that day.

[0070] "Supplementary information on product descriptions and limited-time sales" refers to audio recordings of detailed product descriptions and sales information.

[0071] "Image data" refers to image files of products photographed by each store in the shopping mall using a terminal.

[0072] "Audio data" refers to audio files containing product descriptions and sales information recorded by each store in the shopping mall using a terminal.

[0073] A "server" is a central computer system that receives data sent from terminals and performs analysis and information aggregation.

[0074] "Image recognition" is a technology used by the server to extract product name and price information from image data.

[0075] "Speech recognition" is a technology used by the server to convert voice data into text data and extract the sale times and other supplemental information from it.

[0076] "Flyers in electronic document format" are advertising materials that can be saved and displayed as electronic data, such as PDF or image formats.

[0077] A "pre-registered user" is a consumer who has registered in advance with the system to receive information about the shopping mall.

[0078] "Electronic distribution" is the means of sending flyers and other information to users' devices over the internet.

[0079] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0080] Information collection via devices

[0081] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." For example, the terminal is equipped with a high-resolution camera and voice recording function, allowing a store clerk to take a picture of a fresh tomato and record a voice explanation such as "This tomato is on sale for today only." This terminal can be used with a regular smartphone or tablet. The terminal's application has also been developed as software specifically for the store, making it easy to take pictures and record voice. The terminal saves this data locally and then sends it to the server.

[0082] Send data from the device to the server

[0083] The device application sends the stored image and audio data to the server via Wi-Fi or mobile data networks. Specifically, the device application sends the collected data to the server's API endpoint, and a confirmation message is displayed on the device indicating that the data has been sent.

[0084] Data reception and analysis by the server

[0085] The server receives image data and audio data sent from the device. The received image data is analyzed using image recognition libraries such as OpenCV and TensorFlow to extract product names and prices. For example, from an image of fresh tomatoes, the product name "tomato" and the photographed price information can be recognized. The audio data is also converted into text using the Google Cloud Speech-to-Text API, allowing the extraction of time-limited sale dates and other important supplementary information. The progress of the analysis is recorded in the server log, making it possible to check in real time.

[0086] Automatic flyer generation

[0087] The server automatically generates electronic flyers based on the information collected from each store. For example, using the Adobe PDF library or Python's ReportLab library, flyers containing collected product information, prices, and time sale information are generated. The flyers are saved in PDF or image format, and a notification of generation completion is sent to the administrator by email.

[0088] Electronic Delivery to Users

[0089] The server obtains the LINE account information of pre-registered users and distributes the flyer using the LINE API once it has been generated. For example, the server sends an image or PDF of the flyer to the user's LINE account along with the message "Fresh tomatoes on sale today only!" The status of the distribution is displayed in real time on the server's dashboard, allowing users to check the status.

[0090] Specific examples

[0091] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0092] Example prompts for generative AI models

[0093] "There's a special price on fresh tomatoes at the mall. How can you let users know about this?"

[0094] The above is a specific embodiment for carrying out the present invention. This system allows consumers to easily obtain information on more advantageous products, and each store in the shopping mall can efficiently attract customers.

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1:

[0097] Information collection via devices

[0098] Specific explanation: The device is operated by a store clerk. First, the clerk launches the device's camera app and takes a picture of the "Today's Recommended Product." Then, he or she launches a voice recording app and records a description of the product and supplementary information about the limited-time sale.

[0099] Input: Product image, product description and audio information about the sale.

[0100] Output: Image data file, audio data file.

[0101] Specific operation: When a store clerk takes a photo of a product with the device, a "Photo" button appears on the screen. While recording audio, "Recording" is displayed, and a confirmation message is displayed after recording is complete.

[0102] Step 2:

[0103] Send data from the device to the server

[0104] Specifically: The device sends collected image and audio data to the server via Wi-Fi or mobile data network. The device application sends the collected data to the server's API endpoint.

[0105] Input: Image data files and audio data files stored on the device.

[0106] Output: Image data and audio data sent to the server.

[0107] Specific behavior: A progress bar will be displayed while the file is being sent, and once the sending is complete, a pop-up message will appear saying "Sending complete."

[0108] Step 3:

[0109] Data reception and analysis by the server

[0110] Specifically, the server receives image data and audio data sent from the device. It uses image recognition libraries such as OpenCV and TensorFlow to extract product names and prices from the received image data. It also converts the audio data into text using the Google Cloud Speech-to-Text API, from which it extracts the time of the limited-time sale and other important supplementary information.

[0111] Input: Image data and audio data sent to the server.

[0112] Output: Product information (product name, price), voice text information (limited sale time, supplementary information).

[0113] Specific operation: The progress of the analysis is recorded in the server log. For example, log information such as "Image recognition started," "Voice recognition started," and "Data analysis in progress" is displayed in real time.

[0114] Step 4:

[0115] Automatic flyer generation

[0116] Specifics: Automatically generate electronic flyers based on product information, prices, and time sale information collected by the server. Flyers are generated using the Adobe PDF library and Python's ReportLab library.

[0117] Input: Aggregated product information, pricing information, and sale information.

[0118] Output: e-flyer in PDF or image format.

[0119] Specific operation: When the flyer generation is completed, a notification "Flyer generation completed" will be sent to the administrator by email.

[0120] Step 5:

[0121] Electronic Delivery to Users

[0122] Detailed explanation: The server obtains the LINE account information of pre-registered users and distributes flyers generated using the LINE API. An image and PDF of the flyer are sent to the user's LINE account along with the message "Fresh tomatoes on sale today only!"

[0123] Input: Generated e-flyer, pre-registered user's LINE account information.

[0124] Output: A flyer sent to the user's LINE account.

[0125] Specific operation: The sending status of LINE messages is displayed in real time on the server dashboard. For example, the status is displayed as "Started sending," "Sent," or "Failed sending."

[0126] (Application example 1)

[0127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] There is a need for a low-cost, effective method for each store in a shopping district to deliver discount information to consumers. In particular, there is currently no practical system that allows consumers to check discount information and product information in real time while walking around the store, and realizing this would be a challenge in improving the shopping district's overall ability to attract customers.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0130] In this invention, the server includes: a means for a terminal of a participating store to take an image of "Today's Recommended Product," a means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, a means for the server to receive image and audio data transmitted from the terminal, a means for the server to extract product information using image recognition, a means for the server to extract supplemental information from the audio data using voice recognition, a means for the server to aggregate the acquired information of all stores and automatically generate a flyer, a means for the server to electronically distribute the generated flyer to pre-registered users, a means for the smart glasses to display product information in the store in real time, a means for the smart glasses to provide additional information about products based on voice commands, and a means for the smart glasses to navigate to product shelves. This allows consumers to check the latest deals and product information in real time while walking around the store, increasing consumer purchasing motivation and improving the shopping district's overall customer attraction.

[0131] "Participating store terminals" are electronic devices owned and operated by each store in the shopping district, and have the function of taking pictures of "Today's Recommended Products" and recording audio descriptions of products and supplementary information about limited-time sales.

[0132] "Means for taking pictures of today's recommended products" refers to the function that allows store staff to take pictures of today's recommended products using a camera on a smartphone, tablet, or other device.

[0133] "Means for recording audio supplementary information about product descriptions and limited-time sales" refers to a function that allows store staff to use a microphone on a smartphone or tablet device to record audio supplementary information about product descriptions and limited-time sales.

[0134] "Server" refers to the central system that receives image data and audio data sent from the terminal, analyzes and processes this data, and generates discount information.

[0135] "Image recognition" refers to a technology that automatically extracts product information (such as product name and price) from image data received by the server.

[0136] "Speech recognition" refers to the technology that converts voice data received by the server into text and extracts important supplementary information from the text (such as time of sale and special information).

[0137] "Means for automatically generating flyers" refers to the function by which the server automatically creates electronic flyers containing information useful to users based on data obtained from image recognition and voice recognition.

[0138] "Means for electronic distribution to pre-registered users" refers to the function by which the server uses messaging services such as the LINE API to electronically distribute the generated flyer to the devices of pre-registered users.

[0139] "Smart glasses" refers to a wearable device that has a display that the wearer can see and that can display information in real time.

[0140] "Means for displaying product information within the store" refers to the function of the smart glasses to display information about the current location within the store and surrounding products on the display in real time.

[0141] "Means for providing additional information about a product based on a voice command" refers to the ability of the smart glasses to display more information about a product when the wearer requests it using a voice command.

[0142] "Means for navigating to product shelves" refers to the function of the smart glasses to guide the wearer to the desired product shelf based on their current location.

[0143] This invention is a system that enables each store in a shopping mall to distribute discount information to consumers at low cost, and provides product information in real time within the store using smart glasses. Specific embodiments for implementing the invention are described below.

[0144] System Configuration

[0145] The system consists of the following components:

[0146] 1. Store terminal: This refers to a smartphone or tablet device that store staff use to take photos of the "Today's Recommended Products" and record supplementary information via voice.

[0147] 2. Server: Receives image and audio data sent from the store's terminal, analyzes and processes it, and generates an electronic flyer.

[0148] 3. Smart glasses: A wearable device worn by the user that displays real-time store information and provides detailed product information based on voice commands.

[0149] Functional description of each element

[0150] 1. Device Function: The store clerk uses a smartphone or tablet to take pictures of the "Today's Recommended Products." They also use the voice recording function to record product descriptions and limited-time sale information. This data is first saved locally and then sent to the server.

[0151] 2. Server Functions:

[0152] The received image data is then subjected to image recognition using the Google Cloud Vision API, and product name and price information is automatically extracted.

[0153] The received audio data is converted into text using the Google Cloud Speech-to-Text API, from which the sale time and other important supplementary information are extracted.

[0154] Based on information collected from each store, electronic flyers are automatically generated in PDF or image format.

[0155] The generated flyers are distributed electronically to pre-registered users using the LINE API.

[0156] 3. Functions of smart glasses:

[0157] The smart glasses display real-time product information about the surrounding area as the user walks through the store.

[0158] Using voice commands, users can operate the smart glasses to obtain detailed product information.

[0159] The in-store navigation function provides directions to specific product shelves.

[0160] Specific examples

[0161] For example, if a user is looking for "fresh tomatoes," they can put on the smart glasses and walk around the store, which will do the following:

[0162] 1. Information collection: The camera in the smart glasses takes a picture of the tomato shelf, and the microphone records the salesperson saying, "These tomatoes are on sale. They're fresh and delicious."

[0163] 2. Information analysis: The server uses image recognition to extract the information "tomato" and "special price," and uses voice recognition to convert the information "special price" and "fresh" into text.

[0164] 3. Flyer generation: Based on the above information, an electronic flyer is created and distributed to users via LINE with the message "Fresh tomatoes on sale!"

[0165] Prompt Sentence Examples

[0166] An example of a prompt to input to a generative AI model is as follows:

[0167] ---

[0168] You are a smart eyeglasses assistant that provides real-time product information in the store. When the user approaches a specific product on the shelf, the display will show special offers and recommendations for that product.

[0169] Special Offers:

[0170] Product Name: Tomato

[0171] Price: 200 yen

[0172] Sale information: Special price campaign

[0173] Description: Fresh and delicious tomatoes.

[0174] Use this information to create an electronic flyer and distribute it to users.

[0175] ---

[0176] The above is a specific embodiment for carrying out the present invention. This system allows each store in a shopping mall to efficiently distribute advantageous information to consumers, and also allows consumers to obtain useful information in real time.

[0177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0178] Step 1:

[0179] Information gathering

[0180] The device (smartphone or tablet) is operated by a store clerk. First, the clerk takes a picture of the "Today's Recommended Product." This is then saved locally as image data. At the same time, the device records the clerk's voice and saves the product description and supplementary information about the limited-time sale as audio data. This data is later sent to the server.

[0181] Input: Product images and audio recordings by store staff

[0182] Output: Image and audio data stored locally on the device

[0183] Step 2:

[0184] Data transmission

[0185] The device sends the image and audio data stored locally to the server via an internet connection.

[0186] Input: Locally stored image and audio data

[0187] Output: Image data and audio data sent to the server

[0188] Step 3:

[0189] Image Recognition

[0190] The server passes the received image data to the Google Cloud Vision API, which extracts product information from the product image. Specifically, it analyzes information such as the product name and price and obtains this as text data.

[0191] Input: Image data sent to the server

[0192] Output: Extracted product name and price text data

[0193] Step 4:

[0194] Voice Recognition

[0195] The server passes the received audio data to the Google Cloud Speech-to-Text API, converts the audio into text data, and then extracts the sale time and other supplemental information from the text data.

[0196] Input: Audio data sent to the server

[0197] Output: Text data of supplementary information extracted from the audio data

[0198] Step 5:

[0199] Information collection and flyer generation

[0200] The server aggregates the product information and supplementary information extracted in steps 3 and 4 to generate a single electronic flyer in PDF or image format.

[0201] Input: Text data of extracted product information and supplementary information

[0202] Output: Generated e-flyer (PDF or image format)

[0203] Step 6:

[0204] Electronic distribution

[0205] The server distributes the generated electronic flyers to pre-registered users using the LINE API, and users receive the flyers on their smartphones or other devices.

[0206] Input: Generated e-flyer

[0207] Output: Electronic flyer delivered to the user's device

[0208] Step 7:

[0209] Real-time information display

[0210] As users wear smart glasses and walk around the store, product information is displayed in real time on the glasses' display. This is achieved by the smart glasses knowing their current location within the store and receiving information from the server in real time.

[0211] Input: Real-time data from the server and the user's current location

[0212] Output: Product information displayed on the smart glasses display

[0213] Step 8:

[0214] Voice Commands and Navigation

[0215] When a user issues a voice command, it is recorded by the smart glasses' microphone and sent to the server. The server analyzes the voice command and sends detailed information about the corresponding product to the smart glasses. Navigation to specific product shelves is also based on the voice command.

[0216] Input: User voice command

[0217] Output: Product details and navigation information displayed on smart glasses

[0218] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0219] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[0220] 1. Information collection via devices

[0221] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The clerk shows the product to the terminal's camera, takes still and video images, and saves them as image files. Next, the clerk records audio into the terminal explaining the product and supplementary information about the limited-time sale, which is saved as an audio file.

[0222] 2. Information analysis by the server

[0223] The server receives the image and audio data sent from the device. For the received image data, it uses image recognition technology to extract product information (product name, price). For example, it recognizes text in the image and obtains necessary information from the product label and price display. For the received audio data, it uses voice recognition technology to convert the audio data into text and extracts the time of the limited sale and other special information.

[0224] 3. Generate flyers

[0225] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. The server automatically generates an electronic flyer based on this information, appropriately positioning the product name, price, and supplementary information to create an easy-to-understand flyer. This flyer is output in PDF or image format.

[0226] 4. Electronic Delivery

[0227] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can check the latest daily information on their smartphones, tablets, or other devices.

[0228] 5. User Emotion Recognition

[0229] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions from the user's facial expressions when viewing the flyer and from their voice comments about the flyer. It also collects feedback from the user after viewing the flyer and analyzes the emotion data collected.

[0230] 6. Information optimization

[0231] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer delivery. Specifically, it identifies the products and conditions that the user is interested in and adjusts the content of future flyers accordingly. This feedback loop makes it possible to provide information that matches the user's preferences.

[0232] Specific examples

[0233] For example, suppose a store inputs information into a device about a special sale on fresh tomatoes. In this case, the store clerk takes a photo of the tomatoes and their price information, and provides additional information in voice, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a flyer based on the extracted information and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction, allowing the content of future distributions to be optimized more effectively.

[0234] The above is a specific embodiment for carrying out the present invention. This invention allows consumers to easily obtain more advantageous product information, and each store in the shopping district can efficiently attract customers. Furthermore, by analyzing user reactions using an emotion engine, the information provided will constantly evolve to adapt to the user's preferences.

[0235] The processing flow will be explained below.

[0236] Step 1:

[0237] The device takes images and video of the "Today's Recommended Products." The store clerk points the device's camera at the product and takes a photo, saving it as an image file. Next, the store clerk records audio explaining the product and supplementary information about the limited-time sale, saving it as an audio file.

[0238] Step 2:

[0239] The device sends the captured image and audio files to the server. Specifically, the device sends the data to the server using an HTTP POST request. This request includes the store ID, image file, and audio file.

[0240] Step 3:

[0241] The server analyzes the image data received from the device. The server uses image recognition technology to extract product names, prices, and other relevant information. For example, the server recognizes text in the image and obtains the necessary information from product labels and price displays.

[0242] Step 4:

[0243] The server analyzes the voice data received from the device. The server converts the voice data into text using voice recognition technology, and extracts the time of the limited-time sale and special promotional information from the text. This information is recorded in conjunction with the product information.

[0244] Step 5:

[0245] The server aggregates the product information and supplementary information collected from all stores, compiles the data from each store in a unified format, and organizes it as information for the entire shopping district.

[0246] Step 6:

[0247] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create an easy-to-understand flyer. This flyer is output in PDF format or image format.

[0248] Step 7:

[0249] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, the server uses the LINE API to distribute it to each user. Users receive the flyer on their smartphones, tablets, or other devices.

[0250] Step 8:

[0251] The server uses an emotion engine to analyze the user's browsing behavior. Specifically, it collects facial expressions and voice comments while the user is browsing the flyer and analyzes the emotion data.

[0252] Step 9:

[0253] The server stores the user's emotional data recognized by the emotion engine, and refers to past emotional data to optimize the content of future flyer deliveries to suit the user's preferences.

[0254] The above steps complete the process in which the device collects information, the server analyzes and aggregates the information, generates flyers, and delivers them to users. Furthermore, an emotion engine is used to analyze user reactions, and the information provided constantly evolves to adapt to the user's preferences.

[0255] Example 2

[0256] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] There is a need for a system that allows each store in a shopping district to effectively distribute discount information to consumers at low cost, and also analyzes user reactions to optimize the content of the next distribution. However, currently, there are limited means to efficiently perform these tasks, and it is particularly difficult to analyze user reactions and reflect them in the next distribution. Therefore, in order to solve these problems, the present invention aims to provide a system that easily collects, distributes, and analyzes information, and enables continuous optimization.

[0258] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0259] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, means for the server to receive image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate flyers, means for the server to electronically distribute the generated flyers to pre-registered users, means for the server to analyze users' reactions while viewing the flyers, and means for the server to optimize the content of the next flyer based on the analysis results. This enables each store in the shopping district to distribute discount information efficiently at low cost and further analyze user reactions and reflect them in future flyers.

[0260] "Terminal" refers to a device operated by a staff member at a participating store, and is a device that has the function of recording and transmitting images and audio.

[0261] The "server" is a central system that receives collected image data and audio data, analyzes and processes them, and distributes the information.

[0262] "Image recognition" is a technology that allows a terminal or server to automatically extract specific information (for example, product name or price) from captured image data.

[0263] "Speech recognition" is a technology that analyzes voice data and extracts text data and information.

[0264] "Product information" refers to basic data related to a product, such as the product name and price.

[0265] "Supplementary information" is additional information that accompanies basic product information, such as time of sale or special information for a specific product.

[0266] A "flyer" is an advertising medium that is generated by aggregating product information and supplementary information collected from each store, and is output in PDF or image format.

[0267] "Electronic distribution" refers to the act of sending the generated flyer in digital form to pre-registered users.

[0268] The "emotion engine" is a system that analyzes emotions based on a user's facial expressions and voice comments, and recognizes specific emotional states.

[0269] "Optimization" is the process of adjusting the content of the next flyer based on the analysis results and providing information that matches the user's preferences and reactions.

[0270] "Reaction analysis" is a technology that analyzes the behavior and facial expressions of users when they view electronic flyers to determine their emotions and interests.

[0271] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[0272] Information collection via devices

[0273] The terminal is operated by a store clerk and can be a portable device such as a smartphone or tablet. The clerk uses the device's camera to take images or videos of "Today's Recommended Products" and saves them as image files. Next, the clerk uses the device's microphone to record audio descriptions of the products and supplementary information about the limited-time sale, which is saved as an audio file. For example, a clerk can take a picture of fresh tomatoes and say, "Today's limited-time sale, from 3:00 to 5:00 PM."

[0274] Server-based information analysis

[0275] The server receives the image and audio data sent from the device. It uses image recognition technology such as OpenCV or TensorFlow to extract product information (product name, price) from the received image data. For example, it analyzes the text in the image and obtains the necessary information from the product label and price display. It also converts the audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the time of the limited-time sale and other special information.

[0276] Generate flyers

[0277] The server aggregates product information and supplementary information collected from all stores. It compiles the data from each store in a unified format and organizes it as information for the entire shopping district. For example, it automatically generates electronic flyers based on the data from each store, appropriately positioning product names, prices, and sale information. The generated electronic flyers are output in PDF or image format.

[0278] Electronic distribution

[0279] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can easily check the latest daily information on their smartphones or tablets.

[0280] User Emotion Recognition

[0281] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions by analyzing the user's facial expressions and voice comments when they view the flyer. For example, if a user smiles when they view the flyer or leaves an interesting voice comment, that information is collected as emotion data.

[0282] Information Optimization

[0283] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer distribution. This allows the server to identify products and information that users are interested in and customize the content of the next flyer. For example, if a particular product received a good response in the previous distribution, the server can prioritize the distribution of information similar to that product.

[0284] Examples and prompts

[0285] For example, when a store inputs information that "fresh tomatoes are on sale at a special price," the store clerk takes a photo of the tomato and records the price information. Additionally, a voice message provides additional information, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a digital flyer based on the aggregated data and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction and optimizes the content of future deliveries.

[0286] An example of a prompt is as follows:

[0287] "We collect image and audio data of the 'Today's Recommended Products' provided by each store in the shopping district, and use image and audio recognition technology to extract product information and limited-time sale information. We then compile all of the information into a unified format to generate digital flyers and distribute them to users using the LINE API. We analyze users' emotional reactions and incorporate feedback into the next distribution."

[0288] The above is a specific embodiment for carrying out the present invention. This system enables each store in a shopping mall to efficiently distribute discount information and analyze user reactions to optimize the content of future distributions.

[0289] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0290] Step 1:

[0291] Information gathering and input data generation

[0292] Using the device's camera, the store clerk takes a picture of the "Today's Recommended Product" and saves it as an image file. Similarly, using the device's microphone, the store clerk records audio of the product description and supplementary information about the limited-time sale, and saves it as an audio file. In actual operation, for example, a store clerk can take a picture of a fresh tomato and explain in audio, "Today's limited-time sale, from 3:00 PM to 5:00 PM." The input for this step is the product image and audio of the product description, and the output is an image file and an audio file.

[0293] Step 2:

[0294] Sending and receiving data to the server

[0295] The device sends the generated image and audio files to the server. The server receives these data and prepares them for analysis. The input in this step is the data sent from the device (image and audio files), and the output is these files stored on the server.

[0296] Step 3:

[0297] Processing image data and extracting product information

[0298] The server uses image recognition technology to extract product information (product name, price) from the received image file. For example, it uses tools such as OpenCV or TensorFlow to analyze the text in the image and obtain the necessary information from the product label and price display. The input for this process is the image file, and the output is the extracted product name and price information.

[0299] Step 4:

[0300] Processing audio data and extracting supplementary information

[0301] The server converts the received audio file into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the sale time and other special information. The input in this step is the audio file, and the output is the text supplementary information.

[0302] Step 5:

[0303] Generate electronic flyers

[0304] The server aggregates the product information and supplementary information collected from all stores and automatically generates an electronic flyer. Specifically, it organizes the collected data into a unified format and arranges the product name, price, and supplementary information in an appropriate layout. The input in this step is the product information and supplementary information collected from each store, and the output is an electronic flyer in PDF or image format.

[0305] Step 6:

[0306] Electronic distribution

[0307] The server sends the generated e-flyer to the user's pre-registered LINE account. It distributes it to the user's smartphone or tablet using the LINE API. The input in this step is the generated e-flyer, and the output is the e-flyer distributed to the user's device.

[0308] Step 7:

[0309] User sentiment analysis

[0310] The server uses an emotion engine to analyze the user's reactions while they are viewing the electronic flyer. Specifically, it analyzes the user's facial expressions and voice comments to obtain emotional information. The input in this step is the user's viewing data (facial expressions, voice comments), and the output is the analyzed emotional data.

[0311] Step 8:

[0312] Information Optimization

[0313] The server saves the emotion data obtained by the emotion engine and reflects it in future flyer content. It analyzes past data to identify products and information that the user was particularly interested in, and customizes the content of the next flyer based on that. The input in this step is the analyzed emotion data, and the output is an optimized version of the next digital flyer.

[0314] The above is the specific flow of the program processing of this system.

[0315] (Application example 2)

[0316] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] Traditional shopping districts and retail stores lacked efficient methods for advertising discount information and seasonal products, resulting in insufficient communication of information to consumers and challenges in promoting store visits and sales. Furthermore, there was a lack of methods for collecting and analyzing consumer responses in real time and reflecting them in the next marketing strategy, making it difficult to respond quickly to consumer needs. This made it difficult to effectively attract customers amid intensifying competition between stores.

[0318] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0319] In this invention, the server includes a means for the terminal to capture and record image and audio data, a means for analyzing the data received by the server, a means for analyzing user reactions using an emotion engine, and a means for optimizing the content of the next flyer. This enables shopping malls and retail stores to efficiently distribute information to consumers and analyze their reactions, enabling them to respond quickly to consumer needs.

[0320] "Participating store terminal" refers to equipment used by each participating store to take product images and record audio.

[0321] "Today's Recommended Products" are products that are especially recommended on that day, and refer to products or services that the store wants to promote in a special way.

[0322] "Means for taking images" refers to the function of taking images of products using a device with a camera function and saving them as data.

[0323] "Means of recording audio" refers to the ability to record product descriptions and supplementary information about limited-time sales using a device with a microphone function and save it as audio data.

[0324] A "server" is a central processing unit that analyzes image and audio data and performs various processes.

[0325] The "means for receiving" refers to the function of receiving data transmitted from a terminal.

[0326] "Image recognition" refers to the technology of automatically analyzing and extracting characters and objects from image data.

[0327] "Product information" refers to data that indicates the product name, price, and other related information.

[0328] "Speech recognition" refers to the technology of converting voice data into text data and analyzing it.

[0329] "Additional Information" refers to sale times and other special information.

[0330] "Means for automatically generating flyers" refers to a function that automatically generates visually appealing flyers based on collected data.

[0331] "Means of electronic distribution" refers to the ability to distribute the generated flyer via email or messaging apps.

[0332] "User" refers to a consumer who wishes to receive electronic flyers and has registered in advance.

[0333] "Emotion engine" refers to technology that analyzes emotions from users' facial expressions and feedback comments.

[0334] "Means for analyzing reactions" refers to a function that uses an emotion engine to analyze the reactions of users when they view a flyer.

[0335] "Optimization means" refers to the function of adjusting the content of future flyers based on analyzed response data, and providing optimal information tailored to consumer preferences.

[0336] The present invention is a system for enabling shopping malls and retail stores to efficiently distribute discount information to consumers and analyze their reactions. The system includes a terminal, a server, and an emotion engine for analyzing user reactions.

[0337] 1. Information collection via devices

[0338] The terminal is a device operated by a store clerk at a participating store, and can be a smartphone or tablet. The clerk uses the device's camera to take pictures of the "Today's Recommended Products" and also records audio descriptions of the products and supplementary information about the limited-time sale. This data is temporarily stored on the terminal and later sent to the server.

[0339] 2. Information analysis by the server

[0340] The server receives the image and audio data sent from the device and performs the main analysis process. The image data is converted into text using image recognition software (e.g., Amazon Rekognition) to extract product name and price information. Similarly, the audio data is converted into text using speech recognition software (e.g., Google Cloud Speech-to-Text) to extract limited-time sale times and other special information.

[0341] 3. Generate flyers

[0342] The server aggregates product information and supplemental information collected from all stores and automatically generates visually appealing digital flyers using template generation tools such as Adobe InDesign and Canva API. The generated flyers are saved in PDF format.

[0343] 4. Electronic Delivery

[0344] The server then distributes the generated digital flyers to the LINE accounts of pre-registered users, using the LINE Messaging API, allowing users to check the latest information on their smartphones, tablets, or other devices.

[0345] 5. User Emotion Recognition

[0346] The server uses an emotion engine (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer) to analyze the user's reaction when viewing the flyer. It analyzes the user's facial expressions and feedback comments to recognize positive, negative, and neutral emotions.

[0347] 6. Information optimization

[0348] The server then optimizes the content of the next flyer based on the analyzed emotion data. This allows the display of product information that the user is interested in to be prioritized, improving advertising effectiveness. For example, if the user shows interest in fresh tomatoes, the next flyer will prominently feature information about fresh vegetables and special offers.

[0349] Specific examples

[0350] For example, if a store is selling fresh tomatoes at a special price, a store clerk can take a picture of the tomato with their smartphone and record a voice description of the product, saying, "Today only, special price on sale from 3:00 PM to 5:00 PM." This data is sent to a server and analyzed. The generated flyer is then sent to users via LINE, and the emotion engine analyzes their reactions. As a result, the content of the next flyer can be optimized more effectively.

[0351] Prompt Sentence Examples

[0352] Image analysis prompt:

[0353] Image file name: store_product.jpg

[0354] API used: Image recognition software

[0355] Information to be recognized: Product name, price

[0356] Prompt for speech analysis:

[0357] Audio file name: product_description.wav

[0358] API used: Speech recognition software

[0359] Extracted information: Time of sale, special information

[0360] Emotion recognition prompt:

[0361] User feedback: "These tomatoes look fresh and delicious."

[0362] API used: Emotion recognition software

[0363] Emotions to extract: positive, negative, neutral

[0364] By combining these technologies, shopping districts and retail stores can provide consumers with discount information at the optimal time, enabling effective marketing.

[0365] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0366] Step 1:

[0367] The device takes a picture of the "Today's Recommended Product." The input is a product image taken using the device's camera function, and the output is saved as an image file in the device's storage. A store clerk takes a picture of the product using the device's camera and saves the image.

[0368] Step 2:

[0369] The device records audio for product descriptions and supplementary information about limited-time sales. The input is audio recorded through a microphone, and the output is saved as an audio file in the device's storage. A store clerk explains the product into the microphone, and the audio is saved.

[0370] Step 3:

[0371] The device sends the captured images and recorded audio data to the server. The input is the image and audio files captured and recorded on the device, and the output is obtained by transferring this data to the server. The device uploads the data to the server via the network.

[0372] Step 4:

[0373] The server analyzes the image data it receives. The input is the image file received by the server, and the output is product information extracted from the text and objects in the image. The server analyzes the image using image recognition software (e.g., Amazon Rekognition) and extracts product name and price information.

[0374] Step 5:

[0375] The server analyzes the received audio data. The input is the audio file received by the server, and the output is supplementary information converted from the audio into text. The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text and analyze it.

[0376] Step 6:

[0377] The server aggregates product information and supplementary information collected from all stores and automatically generates flyers. The input is the analyzed product information and supplementary information, and the output is an electronic flyer. The server uses a template generation tool (e.g., Adobe InDesign, Canva API) to generate flyers based on the acquired data.

[0378] Step 7:

[0379] The server distributes the generated e-flyer to pre-registered users. The input is the generated e-flyer and the user's contact information, and the output is the flyer distributed to the recipient user. The server uses the LINE Messaging API to send the flyer to the user's LINE account.

[0380] Step 8:

[0381] The server analyzes the user's reaction when viewing the flyer. The input is the user's facial expression data and feedback comments, and the output is the user's emotion recognition results. The server analyzes the user's reaction using emotion recognition software (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer).

[0382] Step 9:

[0383] The server optimizes the content of the next flyer based on the analysis results. The input is the analyzed emotion data, and the output is the optimized flyer content. Based on the feedback obtained, the server prioritizes product information that matches the user's preferences and reflects it in the next flyer.

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0387] [Second embodiment]

[0388] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0389] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0394] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0395] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0399] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0400] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0401] 1. Information collection via devices

[0402] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The captured images clearly display the products, making them attractive and easy to understand for consumers. The terminal also has a voice recording function, allowing the clerk to record a voice explanation of the products and supplementary information about the limited-time sale. The terminal stores this data locally and then transmits it to the server.

[0403] 2. Information analysis by the server

[0404] The server receives the image and audio data sent from the terminal. It uses image recognition technology to extract product information (such as product name and price) from the received image data. The server also uses voice recognition technology to convert the audio data into text, from which it extracts the time of limited-time sales and other important supplementary information. This makes it possible to integrate and consolidate information for the entire shopping district, rather than just for individual stores.

[0405] 3. Generate flyers

[0406] The server automatically generates electronic flyers based on the information collected from each store. These flyers contain product names, prices, and time sale information, and are laid out so that users can easily see all the useful information at a glance. These flyers are output as electronic documents in PDF or image format.

[0407] 4. Electronic Delivery

[0408] The server obtains the LINE account information of pre-registered users. Once the flyer is generated, it is distributed using the LINE API. Users can receive daily updated discount information on their smartphones, tablets, and other devices. This allows stores in shopping districts to deliver the latest information directly to consumers without having to use large-scale advertising methods.

[0409] Specific examples

[0410] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0411] The above is a specific embodiment for carrying out the present invention. The present invention allows consumers to easily obtain more advantageous product information, and each store in the shopping mall can efficiently attract customers.

[0412] The processing flow will be explained below.

[0413] Step 1:

[0414] The device takes images and videos of "Today's Recommended Products." The store clerk takes still and video images of the products shown to the device's camera, which are saved as image files. Next, the store clerk records audio into the device explaining the products and supplementary information about the limited-time sale, which is saved as an audio file.

[0415] Step 2:

[0416] The device sends the captured image and audio files to the server. Specifically, the data is sent using an HTTP POST request. This request includes the store ID, image file, and audio file, and the server temporarily stores them.

[0417] Step 3:

[0418] The server analyzes the image data received from the device and uses image recognition technology to extract product names, prices, and other relevant information, for example by recognizing text in the image and obtaining the necessary information from product labels and price displays.

[0419] Step 4:

[0420] The server analyzes the voice data received from the device. It uses voice recognition technology to convert the voice data into text. From the converted text, it extracts supplemental information such as time of sale and special promotional content. This information is recorded in conjunction with the product information.

[0421] Step 5:

[0422] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. This allows information from multiple stores to be aggregated into a single flyer.

[0423] Step 6:

[0424] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create a visually easy-to-understand flyer. This flyer is output as a PDF or image file.

[0425] Step 7:

[0426] The server obtains the LINE account information of pre-registered users. It then distributes flyers generated using the LINE API to each user. Users can receive the flyers on their smartphones, tablets, or other devices and check the latest daily information.

[0427] The above steps complete a series of processes in which the terminal collects information, the server analyzes and aggregates the information, generates a flyer, and distributes it to the user.

[0428] Example 1

[0429] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0430] Currently, there is a lack of means for each store in a shopping district to efficiently distribute discount information to consumers at low cost. Conventional advertising methods require large amounts of advertising expenses, which places a heavy burden on small stores in shopping districts. In addition, there are limited means of providing consumers with the latest information in a timely manner. This creates challenges for the shopping district as a whole in attracting customers and communicating with consumers.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio for product descriptions and supplemental information about limited-time sales, means for the terminal to transmit image data and audio data collected by the terminal to the server, means for the server to receive the image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate a flyer in electronic document format, and means for electronically distributing the flyer generated by the server to pre-registered users. This enables each store in the shopping district to provide consumers with the latest information effectively and at low cost, thereby increasing its customer attraction.

[0433] A "participating store terminal" is a device with image capture and audio recording functions that can be operated by store staff at each store in the shopping district.

[0434] "Image of today's recommended product" is a photograph or image file that shows the visual information of the product that each store in the shopping district wants to particularly promote that day.

[0435] "Supplementary information on product descriptions and limited-time sales" refers to audio recordings of detailed product descriptions and sales information.

[0436] "Image data" refers to image files of products photographed by each store in the shopping mall using a terminal.

[0437] "Audio data" refers to audio files containing product descriptions and sales information recorded by each store in the shopping mall using a terminal.

[0438] A "server" is a central computer system that receives data sent from terminals and performs analysis and information aggregation.

[0439] "Image recognition" is a technology used by the server to extract product name and price information from image data.

[0440] "Speech recognition" is a technology used by the server to convert voice data into text data and extract sale times and other supplemental information from it.

[0441] "Flyers in electronic document format" are advertising materials that can be saved and displayed as electronic data, such as PDF or image formats.

[0442] A "pre-registered user" is a consumer who has registered in advance with the system to receive information about the shopping mall.

[0443] "Electronic distribution" is the means of sending flyers and other information to users' devices over the internet.

[0444] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0445] Information collection via devices

[0446] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." For example, the terminal is equipped with a high-resolution camera and voice recording function, allowing a store clerk to take a picture of a fresh tomato and record a voice explanation such as "This tomato is on sale for today only." This terminal can be used with a regular smartphone or tablet. The terminal's application has also been developed as software specifically for the store, making it easy to take pictures and record voice. The terminal saves this data locally and then sends it to the server.

[0447] Send data from the device to the server

[0448] The device application sends the stored image and audio data to the server via Wi-Fi or mobile data networks. Specifically, the device application sends the collected data to the server's API endpoint, and a confirmation message is displayed on the device indicating that the data has been sent.

[0449] Data reception and analysis by the server

[0450] The server receives image data and audio data sent from the device. The received image data is analyzed using image recognition libraries such as OpenCV and TensorFlow to extract product names and prices. For example, from an image of fresh tomatoes, the product name "tomato" and the photographed price information can be recognized. The audio data is also converted into text using the Google Cloud Speech-to-Text API, allowing the extraction of time-limited sale dates and other important supplementary information. The progress of the analysis is recorded in the server log, making it possible to check in real time.

[0451] Automatic flyer generation

[0452] The server automatically generates electronic flyers based on the information collected from each store. For example, using the Adobe PDF library or Python's ReportLab library, flyers containing collected product information, prices, and time sale information are generated. The flyers are saved in PDF or image format, and a notification of generation completion is sent to the administrator by email.

[0453] Electronic Delivery to Users

[0454] The server obtains the LINE account information of pre-registered users and distributes the flyer using the LINE API once it has been generated. For example, the server sends an image or PDF of the flyer to the user's LINE account along with the message "Fresh tomatoes on sale today only!" The status of the distribution is displayed in real time on the server's dashboard, allowing users to check the status.

[0455] Specific examples

[0456] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0457] Example prompts for generative AI models

[0458] "There's a special price on fresh tomatoes at the mall. How can you let users know about this?"

[0459] The above is a specific embodiment for carrying out the present invention. This system allows consumers to easily obtain information on more advantageous products, and each store in the shopping mall can efficiently attract customers.

[0460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0461] Step 1:

[0462] Information collection via devices

[0463] Specific explanation: The device is operated by a store clerk. First, the clerk launches the device's camera app and takes a picture of the "Today's Recommended Product." Then, he or she launches a voice recording app and records a description of the product and supplementary information about the limited-time sale.

[0464] Input: Product image, product description and audio information about the sale.

[0465] Output: Image data file, audio data file.

[0466] Specific operation: When a store clerk takes a photo of a product with the device, a "Photo" button appears on the screen. While recording audio, "Recording" is displayed, and a confirmation message is displayed after recording is complete.

[0467] Step 2:

[0468] Send data from the device to the server

[0469] Specifically: The device sends collected image and audio data to the server via Wi-Fi or mobile data network. The device application sends the collected data to the server's API endpoint.

[0470] Input: Image data files and audio data files stored on the device.

[0471] Output: Image data and audio data sent to the server.

[0472] Specific behavior: A progress bar will be displayed while the file is being sent, and once the sending is complete, a pop-up message will appear saying "Sending complete."

[0473] Step 3:

[0474] Data reception and analysis by the server

[0475] Specifically, the server receives image data and audio data sent from the device. It uses image recognition libraries such as OpenCV and TensorFlow to extract product names and prices from the received image data. It also converts the audio data into text using the Google Cloud Speech-to-Text API, from which it extracts the time of the limited-time sale and other important supplementary information.

[0476] Input: Image data and audio data sent to the server.

[0477] Output: Product information (product name, price), voice text information (limited sale time, supplementary information).

[0478] Specific operation: The progress of the analysis is recorded in the server log. For example, log information such as "Image recognition started," "Voice recognition started," and "Data analysis in progress" is displayed in real time.

[0479] Step 4:

[0480] Automatic flyer generation

[0481] Specifics: Automatically generate electronic flyers based on product information, prices, and time sale information collected by the server. Flyers are generated using the Adobe PDF library and Python's ReportLab library.

[0482] Input: Aggregated product information, pricing information, and sale information.

[0483] Output: e-flyer in PDF or image format.

[0484] Specific operation: When the flyer generation is completed, a notification "Flyer generation completed" will be sent to the administrator by email.

[0485] Step 5:

[0486] Electronic Delivery to Users

[0487] Detailed explanation: The server obtains the LINE account information of pre-registered users and distributes flyers generated using the LINE API. An image and PDF of the flyer are sent to the user's LINE account along with the message "Fresh tomatoes on sale today only!"

[0488] Input: Generated e-flyer, pre-registered user's LINE account information.

[0489] Output: A flyer sent to the user's LINE account.

[0490] Specific operation: The sending status of LINE messages is displayed in real time on the server dashboard. For example, the status is displayed as "Started sending," "Sent," or "Failed sending."

[0491] (Application example 1)

[0492] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0493] There is a need for a low-cost, effective method for each store in a shopping district to deliver discount information to consumers. In particular, there is currently no practical system that allows consumers to check discount information and product information in real time while walking around the store, and realizing this would be a challenge in improving the shopping district's overall ability to attract customers.

[0494] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0495] In this invention, the server includes: a means for a terminal of a participating store to take an image of "Today's Recommended Product," a means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, a means for the server to receive image and audio data transmitted from the terminal, a means for the server to extract product information using image recognition, a means for the server to extract supplemental information from the audio data using voice recognition, a means for the server to aggregate the acquired information of all stores and automatically generate a flyer, a means for the server to electronically distribute the generated flyer to pre-registered users, a means for the smart glasses to display product information in the store in real time, a means for the smart glasses to provide additional information about products based on voice commands, and a means for the smart glasses to navigate to product shelves. This allows consumers to check the latest deals and product information in real time while walking around the store, increasing consumer purchasing motivation and improving the shopping district's overall customer attraction.

[0496] "Participating store terminals" are electronic devices owned and operated by each store in the shopping district, and have the function of taking pictures of "Today's Recommended Products" and recording audio descriptions of products and supplementary information about limited-time sales.

[0497] "Means for taking pictures of today's recommended products" refers to the function that allows store staff to take pictures of today's recommended products using a camera on a smartphone, tablet, or other device.

[0498] "Means for recording audio supplementary information about product descriptions and limited-time sales" refers to a function that allows store staff to use a microphone on a smartphone or tablet device to record audio supplementary information about product descriptions and limited-time sales.

[0499] "Server" refers to the central system that receives image data and audio data sent from the terminal, analyzes and processes this data, and generates discount information.

[0500] "Image recognition" refers to a technology that automatically extracts product information (such as product name and price) from image data received by the server.

[0501] "Speech recognition" refers to the technology that converts voice data received by the server into text and extracts important supplementary information from the text (such as time of sale and special information).

[0502] "Means for automatically generating flyers" refers to the function by which the server automatically creates electronic flyers containing information useful to users based on data obtained from image recognition and voice recognition.

[0503] "Means for electronic distribution to pre-registered users" refers to the function by which the server uses messaging services such as the LINE API to electronically distribute the generated flyer to the devices of pre-registered users.

[0504] "Smart glasses" refers to a wearable device that has a display that the wearer can see and that can display information in real time.

[0505] "Means for displaying product information within the store" refers to the function of the smart glasses to display information about the current location within the store and surrounding products on the display in real time.

[0506] "Means for providing additional information about a product based on a voice command" refers to the ability of the smart glasses to display more information about a product when the wearer requests it using a voice command.

[0507] "Means for navigating to product shelves" refers to the function of the smart glasses to guide the wearer to the desired product shelf based on their current location.

[0508] This invention is a system that enables each store in a shopping mall to distribute discount information to consumers at low cost, and provides product information in real time within the store using smart glasses. Specific embodiments for implementing the invention are described below.

[0509] System Configuration

[0510] The system consists of the following components:

[0511] 1. Store terminal: This refers to a smartphone or tablet device that store staff use to take photos of the "Today's Recommended Products" and record supplementary information via voice.

[0512] 2. Server: Receives image and audio data sent from the store's terminal, analyzes and processes it, and generates an electronic flyer.

[0513] 3. Smart glasses: A wearable device worn by the user that displays real-time store information and provides detailed product information based on voice commands.

[0514] Functional description of each element

[0515] 1. Device Function: The store clerk uses a smartphone or tablet to take pictures of the "Today's Recommended Products." They also use the voice recording function to record product descriptions and limited-time sale information. This data is first saved locally and then sent to the server.

[0516] 2. Server Functions:

[0517] The received image data is then subjected to image recognition using the Google Cloud Vision API, and product name and price information is automatically extracted.

[0518] The received audio data is converted into text using the Google Cloud Speech-to-Text API, from which the sale time and other important supplementary information are extracted.

[0519] Based on information collected from each store, electronic flyers are automatically generated in PDF or image format.

[0520] The generated flyers are distributed electronically to pre-registered users using the LINE API.

[0521] 3. Functions of smart glasses:

[0522] The smart glasses display real-time product information about the surrounding area as the user walks through the store.

[0523] Using voice commands, users can operate the smart glasses to obtain detailed product information.

[0524] The in-store navigation function provides directions to specific product shelves.

[0525] Specific examples

[0526] For example, if a user is looking for "fresh tomatoes," they can put on the smart glasses and walk around the store, which will do the following:

[0527] 1. Information collection: The camera in the smart glasses takes a picture of the tomato shelf, and the microphone records the salesperson saying, "These tomatoes are on sale. They're fresh and delicious."

[0528] 2. Information analysis: The server uses image recognition to extract the information "tomato" and "special price," and uses voice recognition to convert the information "special price" and "fresh" into text.

[0529] 3. Flyer generation: Based on the above information, an electronic flyer is created and distributed to users via LINE with the message "Fresh tomatoes on sale!"

[0530] Prompt Sentence Examples

[0531] An example of a prompt to input to a generative AI model is as follows:

[0532] ---

[0533] You are a smart eyeglasses assistant that provides real-time product information in the store. When the user approaches a specific product on the shelf, the display will show special offers and recommendations for that product.

[0534] Special Offers:

[0535] Product Name: Tomato

[0536] Price: 200 yen

[0537] Sale information: Special price campaign

[0538] Description: Fresh and delicious tomatoes.

[0539] Use this information to create an electronic flyer and distribute it to users.

[0540] ---

[0541] The above is a specific embodiment for carrying out the present invention. This system allows each store in a shopping mall to efficiently distribute advantageous information to consumers, and also allows consumers to obtain useful information in real time.

[0542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0543] Step 1:

[0544] Information gathering

[0545] The device (smartphone or tablet) is operated by a store clerk. First, the clerk takes a picture of the "Today's Recommended Product." This is then saved locally as image data. At the same time, the device records the clerk's voice and saves the product description and supplementary information about the limited-time sale as audio data. This data is later sent to the server.

[0546] Input: Product images and audio recordings by store staff

[0547] Output: Image and audio data stored locally on the device

[0548] Step 2:

[0549] Data transmission

[0550] The device sends the image and audio data stored locally to the server via an internet connection.

[0551] Input: Locally stored image and audio data

[0552] Output: Image data and audio data sent to the server

[0553] Step 3:

[0554] Image Recognition

[0555] The server passes the received image data to the Google Cloud Vision API, which extracts product information from the product image. Specifically, it analyzes information such as the product name and price and obtains this as text data.

[0556] Input: Image data sent to the server

[0557] Output: Extracted product name and price text data

[0558] Step 4:

[0559] Voice Recognition

[0560] The server passes the received audio data to the Google Cloud Speech-to-Text API, converts the audio into text data, and then extracts the sale time and other supplemental information from the text data.

[0561] Input: Audio data sent to the server

[0562] Output: Text data of supplementary information extracted from the audio data

[0563] Step 5:

[0564] Information collection and flyer generation

[0565] The server aggregates the product information and supplementary information extracted in steps 3 and 4 to generate a single electronic flyer in PDF or image format.

[0566] Input: Text data of extracted product information and supplementary information

[0567] Output: Generated e-flyer (PDF or image format)

[0568] Step 6:

[0569] Electronic distribution

[0570] The server distributes the generated electronic flyers to pre-registered users using the LINE API, and users receive the flyers on their smartphones or other devices.

[0571] Input: Generated e-flyer

[0572] Output: Electronic flyer delivered to the user's device

[0573] Step 7:

[0574] Real-time information display

[0575] As users wear smart glasses and walk around the store, product information is displayed in real time on the glasses' display. This is achieved by the smart glasses knowing their current location within the store and receiving information from the server in real time.

[0576] Input: Real-time data from the server and the user's current location

[0577] Output: Product information displayed on the smart glasses display

[0578] Step 8:

[0579] Voice Commands and Navigation

[0580] When a user issues a voice command, it is recorded by the smart glasses' microphone and sent to the server. The server analyzes the voice command and sends detailed information about the corresponding product to the smart glasses. Navigation to specific product shelves is also based on the voice command.

[0581] Input: User voice command

[0582] Output: Product details and navigation information displayed on smart glasses

[0583] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0584] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[0585] 1. Information collection via devices

[0586] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The clerk shows the product to the terminal's camera, takes still and video images, and saves them as image files. Next, the clerk records audio into the terminal explaining the product and supplementary information about the limited-time sale, which is saved as an audio file.

[0587] 2. Information analysis by the server

[0588] The server receives the image and audio data sent from the device. For the received image data, it uses image recognition technology to extract product information (product name, price). For example, it recognizes text in the image and obtains necessary information from the product label and price display. For the received audio data, it uses voice recognition technology to convert the audio data into text and extracts the time of the limited sale and other special information.

[0589] 3. Generate flyers

[0590] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. The server automatically generates an electronic flyer based on this information, appropriately positioning the product name, price, and supplementary information to create an easy-to-understand flyer. This flyer is output in PDF or image format.

[0591] 4. Electronic Delivery

[0592] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can check the latest daily information on their smartphones, tablets, or other devices.

[0593] 5. User Emotion Recognition

[0594] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions from the user's facial expressions when viewing the flyer and from their voice comments about the flyer. It also collects feedback from the user after viewing the flyer and analyzes the emotion data collected.

[0595] 6. Information optimization

[0596] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer delivery. Specifically, it identifies the products and conditions that the user is interested in and adjusts the content of future flyers accordingly. This feedback loop makes it possible to provide information that matches the user's preferences.

[0597] Specific examples

[0598] For example, suppose a store inputs information into a device about a special sale on fresh tomatoes. In this case, the store clerk takes a photo of the tomatoes and their price information, and provides additional information in voice, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a flyer based on the extracted information and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction, allowing the content of future distributions to be optimized more effectively.

[0599] The above is a specific embodiment for carrying out the present invention. This invention allows consumers to easily obtain more advantageous product information, and each store in the shopping district can efficiently attract customers. Furthermore, by analyzing user reactions using an emotion engine, the information provided will constantly evolve to adapt to the user's preferences.

[0600] The processing flow will be explained below.

[0601] Step 1:

[0602] The device takes images and video of the "Today's Recommended Products." The store clerk points the device's camera at the product and takes a photo, saving it as an image file. Next, the store clerk records audio explaining the product and supplementary information about the limited-time sale, saving it as an audio file.

[0603] Step 2:

[0604] The device sends the captured image and audio files to the server. Specifically, the device sends the data to the server using an HTTP POST request. This request includes the store ID, image file, and audio file.

[0605] Step 3:

[0606] The server analyzes the image data received from the device. The server uses image recognition technology to extract product names, prices, and other relevant information. For example, the server recognizes text in the image and obtains the necessary information from product labels and price displays.

[0607] Step 4:

[0608] The server analyzes the voice data received from the device. The server converts the voice data into text using voice recognition technology, and extracts the time of the limited-time sale and special promotional information from the text. This information is recorded in conjunction with the product information.

[0609] Step 5:

[0610] The server aggregates the product information and supplementary information collected from all stores, compiles the data from each store in a unified format, and organizes it as information for the entire shopping district.

[0611] Step 6:

[0612] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create an easy-to-understand flyer. This flyer is output in PDF format or image format.

[0613] Step 7:

[0614] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, the server uses the LINE API to distribute it to each user. Users receive the flyer on their smartphones, tablets, or other devices.

[0615] Step 8:

[0616] The server uses an emotion engine to analyze the user's browsing behavior. Specifically, it collects facial expressions and voice comments while the user is browsing the flyer and analyzes the emotion data.

[0617] Step 9:

[0618] The server stores the user's emotional data recognized by the emotion engine, and refers to past emotional data to optimize the content of future flyer deliveries to suit the user's preferences.

[0619] The above steps complete the process in which the device collects information, the server analyzes and aggregates the information, generates flyers, and delivers them to users. Furthermore, an emotion engine is used to analyze user reactions, and the information provided constantly evolves to adapt to the user's preferences.

[0620] Example 2

[0621] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] There is a need for a system that allows each store in a shopping district to effectively distribute discount information to consumers at low cost, and also analyzes user reactions to optimize the content of the next distribution. However, currently, there are limited means to efficiently perform these tasks, and it is particularly difficult to analyze user reactions and reflect them in the next distribution. Therefore, in order to solve these problems, the present invention aims to provide a system that easily collects, distributes, and analyzes information, and enables continuous optimization.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0624] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, means for the server to receive image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate flyers, means for the server to electronically distribute the generated flyers to pre-registered users, means for the server to analyze users' reactions while viewing the flyers, and means for the server to optimize the content of the next flyer based on the analysis results. This enables each store in the shopping district to distribute discount information efficiently at low cost and further analyze user reactions and reflect them in future flyers.

[0625] "Terminal" refers to a device operated by a staff member at a participating store, and is a device that has the function of recording and transmitting images and audio.

[0626] The "server" is a central system that receives collected image data and audio data, analyzes and processes them, and distributes the information.

[0627] "Image recognition" is a technology that allows a terminal or server to automatically extract specific information (for example, product name or price) from captured image data.

[0628] "Speech recognition" is a technology that analyzes voice data and extracts text data and information.

[0629] "Product information" refers to basic data related to a product, such as the product name and price.

[0630] "Supplementary information" is additional information that accompanies basic product information, such as time of sale or special information for a specific product.

[0631] A "flyer" is an advertising medium that is generated by aggregating product information and supplementary information collected from each store, and is output in PDF or image format.

[0632] "Electronic distribution" refers to the act of sending the generated flyer in digital form to pre-registered users.

[0633] The "emotion engine" is a system that analyzes emotions based on a user's facial expressions and voice comments, and recognizes specific emotional states.

[0634] "Optimization" is the process of adjusting the content of the next flyer based on the analysis results and providing information that matches the user's preferences and reactions.

[0635] "Reaction analysis" is a technology that analyzes the behavior and facial expressions of users when they view electronic flyers to determine their emotions and interests.

[0636] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[0637] Information collection via devices

[0638] The terminal is operated by a store clerk and can be a portable device such as a smartphone or tablet. The clerk uses the device's camera to take images or videos of "Today's Recommended Products" and saves them as image files. Next, the clerk uses the device's microphone to record audio descriptions of the products and supplementary information about the limited-time sale, which is saved as an audio file. For example, a clerk can take a picture of fresh tomatoes and say, "Today's limited-time sale, from 3:00 to 5:00 PM."

[0639] Server-based information analysis

[0640] The server receives the image and audio data sent from the device. It uses image recognition technology such as OpenCV or TensorFlow to extract product information (product name, price) from the received image data. For example, it analyzes the text in the image and obtains the necessary information from the product label and price display. It also converts the audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the time of the limited-time sale and other special information.

[0641] Generate flyers

[0642] The server aggregates product information and supplementary information collected from all stores. It compiles the data from each store in a unified format and organizes it as information for the entire shopping district. For example, it automatically generates electronic flyers based on the data from each store, appropriately positioning product names, prices, and sale information. The generated electronic flyers are output in PDF or image format.

[0643] Electronic distribution

[0644] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can easily check the latest daily information on their smartphones or tablets.

[0645] User Emotion Recognition

[0646] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions by analyzing the user's facial expressions and voice comments when they view the flyer. For example, if a user smiles when they view the flyer or leaves an interesting voice comment, that information is collected as emotion data.

[0647] Information Optimization

[0648] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer distribution. This allows the server to identify products and information that users are interested in and customize the content of the next flyer. For example, if a particular product received a good response in the previous distribution, the server can prioritize the distribution of information similar to that product.

[0649] Examples and prompts

[0650] For example, when a store inputs information that "fresh tomatoes are on sale at a special price," the store clerk takes a photo of the tomato and records the price information. Additionally, a voice message provides additional information, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a digital flyer based on the aggregated data and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction and optimizes the content of future deliveries.

[0651] An example of a prompt is as follows:

[0652] "We collect image and audio data of the 'Today's Recommended Products' provided by each store in the shopping district, and use image and audio recognition technology to extract product information and limited-time sale information. We then compile all of the information into a unified format to generate digital flyers and distribute them to users using the LINE API. We analyze users' emotional reactions and incorporate feedback into the next distribution."

[0653] The above is a specific embodiment for carrying out the present invention. This system enables each store in a shopping mall to efficiently distribute discount information and analyze user reactions to optimize the content of future distributions.

[0654] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0655] Step 1:

[0656] Information gathering and input data generation

[0657] Using the device's camera, the store clerk takes a picture of the "Today's Recommended Product" and saves it as an image file. Similarly, using the device's microphone, the store clerk records audio of the product description and supplementary information about the limited-time sale, and saves it as an audio file. In actual operation, for example, a store clerk can take a picture of a fresh tomato and explain in audio, "Today's limited-time sale, from 3:00 PM to 5:00 PM." The input for this step is the product image and audio of the product description, and the output is an image file and an audio file.

[0658] Step 2:

[0659] Sending and receiving data to the server

[0660] The device sends the generated image and audio files to the server. The server receives these data and prepares them for analysis. The input in this step is the data sent from the device (image and audio files), and the output is these files stored on the server.

[0661] Step 3:

[0662] Processing image data and extracting product information

[0663] The server uses image recognition technology to extract product information (product name, price) from the received image file. For example, it uses tools such as OpenCV or TensorFlow to analyze the text in the image and obtain the necessary information from the product label and price display. The input for this process is the image file, and the output is the extracted product name and price information.

[0664] Step 4:

[0665] Processing audio data and extracting supplementary information

[0666] The server converts the received audio file into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the sale time and other special information. The input in this step is the audio file, and the output is the text supplementary information.

[0667] Step 5:

[0668] Generate electronic flyers

[0669] The server aggregates the product information and supplementary information collected from all stores and automatically generates an electronic flyer. Specifically, it organizes the collected data into a unified format and arranges the product name, price, and supplementary information in an appropriate layout. The input in this step is the product information and supplementary information collected from each store, and the output is an electronic flyer in PDF or image format.

[0670] Step 6:

[0671] Electronic distribution

[0672] The server sends the generated e-flyer to the user's pre-registered LINE account. It distributes it to the user's smartphone or tablet using the LINE API. The input in this step is the generated e-flyer, and the output is the e-flyer distributed to the user's device.

[0673] Step 7:

[0674] User sentiment analysis

[0675] The server uses an emotion engine to analyze the user's reactions while they are viewing the electronic flyer. Specifically, it analyzes the user's facial expressions and voice comments to obtain emotional information. The input in this step is the user's viewing data (facial expressions, voice comments), and the output is the analyzed emotional data.

[0676] Step 8:

[0677] Information Optimization

[0678] The server saves the emotion data obtained by the emotion engine and reflects it in future flyer content. It analyzes past data to identify products and information that the user was particularly interested in, and customizes the content of the next flyer based on that. The input in this step is the analyzed emotion data, and the output is an optimized version of the next digital flyer.

[0679] The above is the specific flow of the program processing of this system.

[0680] (Application example 2)

[0681] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] Traditional shopping districts and retail stores lacked efficient methods for advertising discount information and seasonal products, resulting in insufficient communication of information to consumers and challenges in promoting store visits and sales. Furthermore, there was a lack of methods for collecting and analyzing consumer responses in real time and reflecting them in the next marketing strategy, making it difficult to respond quickly to consumer needs. This made it difficult to effectively attract customers amid intensifying competition between stores.

[0683] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0684] In this invention, the server includes a means for the terminal to capture and record image and audio data, a means for analyzing the data received by the server, a means for analyzing user reactions using an emotion engine, and a means for optimizing the content of the next flyer. This enables shopping malls and retail stores to efficiently distribute information to consumers and analyze their reactions, enabling them to respond quickly to consumer needs.

[0685] "Participating store terminal" refers to equipment used by each participating store to take product images and record audio.

[0686] "Today's Recommended Products" are products that are especially recommended on that day, and refer to products or services that the store wants to promote in a special way.

[0687] "Means for taking images" refers to the function of taking images of products using a device with a camera function and saving them as data.

[0688] "Means of recording audio" refers to the ability to record product descriptions and supplementary information about limited-time sales using a device with a microphone function and save it as audio data.

[0689] A "server" is a central processing unit that analyzes image and audio data and performs various processes.

[0690] The "means for receiving" refers to the function of receiving data transmitted from a terminal.

[0691] "Image recognition" refers to the technology of automatically analyzing and extracting characters and objects from image data.

[0692] "Product information" refers to data that indicates the product name, price, and other related information.

[0693] "Speech recognition" refers to the technology of converting voice data into text data and analyzing it.

[0694] "Additional Information" refers to sale times and other special information.

[0695] "Means for automatically generating flyers" refers to a function that automatically generates visually appealing flyers based on collected data.

[0696] "Means of electronic distribution" refers to the ability to distribute the generated flyer via email or messaging apps.

[0697] "User" refers to a consumer who wishes to receive electronic flyers and has registered in advance.

[0698] "Emotion engine" refers to technology that analyzes emotions from users' facial expressions and feedback comments.

[0699] "Means for analyzing reactions" refers to a function that uses an emotion engine to analyze the reactions of users when they view a flyer.

[0700] "Optimization means" refers to the function of adjusting the content of future flyers based on analyzed response data, and providing optimal information tailored to consumer preferences.

[0701] The present invention is a system for enabling shopping malls and retail stores to efficiently distribute discount information to consumers and analyze their reactions. The system includes a terminal, a server, and an emotion engine for analyzing user reactions.

[0702] 1. Information collection via devices

[0703] The terminal is a device operated by a store clerk at a participating store, and can be a smartphone or tablet. The clerk uses the device's camera to take pictures of the "Today's Recommended Products" and also records audio descriptions of the products and supplementary information about the limited-time sale. This data is temporarily stored on the terminal and later sent to the server.

[0704] 2. Information analysis by the server

[0705] The server receives the image and audio data sent from the device and performs the main analysis process. The image data is converted into text using image recognition software (e.g., Amazon Rekognition) to extract product name and price information. Similarly, the audio data is converted into text using speech recognition software (e.g., Google Cloud Speech-to-Text) to extract limited-time sale times and other special information.

[0706] 3. Generate flyers

[0707] The server aggregates product information and supplemental information collected from all stores and automatically generates visually appealing digital flyers using template generation tools such as Adobe InDesign and Canva API. The generated flyers are saved in PDF format.

[0708] 4. Electronic Delivery

[0709] The server then distributes the generated digital flyers to the LINE accounts of pre-registered users, using the LINE Messaging API, allowing users to check the latest information on their smartphones, tablets, or other devices.

[0710] 5. User Emotion Recognition

[0711] The server uses an emotion engine (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer) to analyze the user's reaction when viewing the flyer. It analyzes the user's facial expressions and feedback comments to recognize positive, negative, and neutral emotions.

[0712] 6. Information optimization

[0713] The server then optimizes the content of the next flyer based on the analyzed emotion data. This allows the display of product information that the user is interested in to be prioritized, improving advertising effectiveness. For example, if the user shows interest in fresh tomatoes, the next flyer will prominently feature information about fresh vegetables and special offers.

[0714] Specific examples

[0715] For example, if a store is selling fresh tomatoes at a special price, a store clerk can take a picture of the tomato with their smartphone and record a voice description of the product, saying, "Today only, special price on sale from 3:00 PM to 5:00 PM." This data is sent to a server and analyzed. The generated flyer is then sent to users via LINE, and the emotion engine analyzes their reactions. As a result, the content of the next flyer can be optimized more effectively.

[0716] Prompt Sentence Examples

[0717] Image analysis prompt:

[0718] Image file name: store_product.jpg

[0719] API used: Image recognition software

[0720] Information to be recognized: Product name, price

[0721] Prompt for speech analysis:

[0722] Audio file name: product_description.wav

[0723] API used: Speech recognition software

[0724] Extracted information: Time of sale, special information

[0725] Emotion recognition prompt:

[0726] User feedback: "These tomatoes look fresh and delicious."

[0727] API used: Emotion recognition software

[0728] Emotions to extract: positive, negative, neutral

[0729] By combining these technologies, shopping districts and retail stores can provide consumers with discount information at the optimal time, enabling effective marketing.

[0730] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0731] Step 1:

[0732] The device takes a picture of the "Today's Recommended Product." The input is a product image taken using the device's camera function, and the output is saved as an image file in the device's storage. A store clerk takes a picture of the product using the device's camera and saves the image.

[0733] Step 2:

[0734] The device records audio for product descriptions and supplementary information about limited-time sales. The input is audio recorded through a microphone, and the output is saved as an audio file in the device's storage. A store clerk explains the product into the microphone, and the audio is saved.

[0735] Step 3:

[0736] The device sends the captured images and recorded audio data to the server. The input is the image and audio files captured and recorded on the device, and the output is obtained by transferring this data to the server. The device uploads the data to the server via the network.

[0737] Step 4:

[0738] The server analyzes the image data it receives. The input is the image file received by the server, and the output is product information extracted from the text and objects in the image. The server analyzes the image using image recognition software (e.g., Amazon Rekognition) and extracts product name and price information.

[0739] Step 5:

[0740] The server analyzes the received audio data. The input is the audio file received by the server, and the output is supplementary information converted from the audio into text. The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text and analyze it.

[0741] Step 6:

[0742] The server aggregates product information and supplementary information collected from all stores and automatically generates flyers. The input is the analyzed product information and supplementary information, and the output is an electronic flyer. The server uses a template generation tool (e.g., Adobe InDesign, Canva API) to generate flyers based on the acquired data.

[0743] Step 7:

[0744] The server distributes the generated e-flyer to pre-registered users. The input is the generated e-flyer and the user's contact information, and the output is the flyer distributed to the recipient user. The server uses the LINE Messaging API to send the flyer to the user's LINE account.

[0745] Step 8:

[0746] The server analyzes the user's reaction when viewing the flyer. The input is the user's facial expression data and feedback comments, and the output is the user's emotion recognition results. The server analyzes the user's reaction using emotion recognition software (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer).

[0747] Step 9:

[0748] The server optimizes the content of the next flyer based on the analysis results. The input is the analyzed emotion data, and the output is the optimized flyer content. Based on the feedback obtained, the server prioritizes product information that matches the user's preferences and reflects it in the next flyer.

[0749] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0750] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0752] [Third embodiment]

[0753] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0754] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0755] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0756] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0757] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0758] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0759] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0760] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0761] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0762] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0763] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0764] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0765] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0766] 1. Information collection via devices

[0767] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The captured images clearly display the products, making them attractive and easy to understand for consumers. The terminal also has a voice recording function, allowing the clerk to record a voice explanation of the products and supplementary information about the limited-time sale. The terminal stores this data locally and then transmits it to the server.

[0768] 2. Information analysis by the server

[0769] The server receives the image and audio data sent from the terminal. It uses image recognition technology to extract product information (such as product name and price) from the received image data. The server also uses voice recognition technology to convert the audio data into text, from which it extracts the time of limited-time sales and other important supplementary information. This makes it possible to integrate and consolidate information for the entire shopping district, rather than just for individual stores.

[0770] 3. Generate flyers

[0771] The server automatically generates electronic flyers based on the information collected from each store. These flyers contain product names, prices, and time sale information, and are laid out so that users can easily see all the useful information at a glance. These flyers are output as electronic documents in PDF or image format.

[0772] 4. Electronic Delivery

[0773] The server obtains the LINE account information of pre-registered users. Once the flyer is generated, it is distributed using the LINE API. Users can receive daily updated discount information on their smartphones, tablets, and other devices. This allows stores in shopping districts to deliver the latest information directly to consumers without having to use large-scale advertising methods.

[0774] Specific examples

[0775] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0776] The above is a specific embodiment for carrying out the present invention. The present invention allows consumers to easily obtain more advantageous product information, and each store in the shopping mall can efficiently attract customers.

[0777] The processing flow will be explained below.

[0778] Step 1:

[0779] The device takes images and videos of "Today's Recommended Products." The store clerk takes still and video images of the products shown to the device's camera, which are saved as image files. Next, the store clerk records audio into the device explaining the products and supplementary information about the limited-time sale, which is saved as an audio file.

[0780] Step 2:

[0781] The device sends the captured image and audio files to the server. Specifically, the data is sent using an HTTP POST request. This request includes the store ID, image file, and audio file, and the server temporarily stores them.

[0782] Step 3:

[0783] The server analyzes the image data received from the device and uses image recognition technology to extract product names, prices, and other relevant information, for example by recognizing text in the image and obtaining the necessary information from product labels and price displays.

[0784] Step 4:

[0785] The server analyzes the voice data received from the device. It uses voice recognition technology to convert the voice data into text. From the converted text, it extracts supplemental information such as time of sale and special promotional content. This information is recorded in conjunction with the product information.

[0786] Step 5:

[0787] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. This allows information from multiple stores to be aggregated into a single flyer.

[0788] Step 6:

[0789] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create a visually easy-to-understand flyer. This flyer is output as a PDF or image file.

[0790] Step 7:

[0791] The server obtains the LINE account information of pre-registered users. It then distributes flyers generated using the LINE API to each user. Users can receive the flyers on their smartphones, tablets, or other devices and check the latest daily information.

[0792] The above steps complete a series of processes in which the terminal collects information, the server analyzes and aggregates the information, generates a flyer, and distributes it to the user.

[0793] Example 1

[0794] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0795] Currently, there is a lack of means for each store in a shopping district to efficiently distribute discount information to consumers at low cost. Conventional advertising methods require large amounts of advertising expenses, which places a heavy burden on small stores in shopping districts. In addition, there are limited means of providing consumers with the latest information in a timely manner. This creates challenges for the shopping district as a whole in attracting customers and communicating with consumers.

[0796] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0797] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio for product descriptions and supplemental information about limited-time sales, means for the terminal to transmit image data and audio data collected by the terminal to the server, means for the server to receive the image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate a flyer in electronic document format, and means for electronically distributing the flyer generated by the server to pre-registered users. This enables each store in the shopping district to provide consumers with the latest information effectively and at low cost, thereby increasing its customer attraction.

[0798] A "participating store terminal" is a device with image capture and audio recording functions that can be operated by store staff at each store in the shopping district.

[0799] "Image of today's recommended product" is a photograph or image file that shows the visual information of the product that each store in the shopping district wants to particularly promote that day.

[0800] "Supplementary information on product descriptions and limited-time sales" refers to audio recordings of detailed product descriptions and sales information.

[0801] "Image data" refers to image files of products photographed by each store in the shopping mall using a terminal.

[0802] "Audio data" refers to audio files containing product descriptions and sales information recorded by each store in the shopping mall using a terminal.

[0803] A "server" is a central computer system that receives data sent from terminals and performs analysis and information aggregation.

[0804] "Image recognition" is a technology used by the server to extract product name and price information from image data.

[0805] "Speech recognition" is a technology used by the server to convert voice data into text data and extract sale times and other supplemental information from it.

[0806] "Flyers in electronic document format" are advertising materials that can be saved and displayed as electronic data, such as PDF or image formats.

[0807] A "pre-registered user" is a consumer who has registered in advance with the system to receive information about the shopping mall.

[0808] "Electronic distribution" is the means of sending flyers and other information to users' devices over the internet.

[0809] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[0810] Information collection via devices

[0811] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." For example, the terminal is equipped with a high-resolution camera and voice recording function, allowing a store clerk to take a picture of a fresh tomato and record a voice explanation such as "This tomato is on sale for today only." This terminal can be used with a regular smartphone or tablet. The terminal's application has also been developed as software specifically for the store, making it easy to take pictures and record voice. The terminal saves this data locally and then sends it to the server.

[0812] Send data from the device to the server

[0813] The device application sends the stored image and audio data to the server via Wi-Fi or a mobile data network. Specifically, the device application sends the collected data to the server's API endpoint, and a confirmation message is displayed on the device indicating that the data has been sent.

[0814] Data reception and analysis by the server

[0815] The server receives image data and audio data sent from the device. The received image data is analyzed using image recognition libraries such as OpenCV and TensorFlow to extract product names and prices. For example, from an image of fresh tomatoes, the product name "tomato" and the photographed price information can be recognized. The audio data is also converted into text using the Google Cloud Speech-to-Text API, allowing the extraction of time-limited sale dates and other important supplementary information. The progress of the analysis is recorded in the server log, making it possible to check in real time.

[0816] Automatic flyer generation

[0817] The server automatically generates electronic flyers based on the information collected from each store. For example, using the Adobe PDF library or Python's ReportLab library, flyers containing collected product information, prices, and time sale information are generated. The flyers are saved in PDF or image format, and a notification of generation completion is sent to the administrator by email.

[0818] Electronic Delivery to Users

[0819] The server obtains the LINE account information of pre-registered users and distributes the flyer using the LINE API once it has been generated. For example, the server sends an image or PDF of the flyer to the user's LINE account along with the message "Fresh tomatoes on sale today only!" The status of the distribution is displayed in real time on the server's dashboard, allowing users to check the status.

[0820] Specific examples

[0821] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[0822] Example prompts for generative AI models

[0823] "There's a special price on fresh tomatoes at the mall. How can you let users know about this?"

[0824] The above is a specific embodiment for carrying out the present invention. This system allows consumers to easily obtain information on more advantageous products, and each store in the shopping mall can efficiently attract customers.

[0825] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0826] Step 1:

[0827] Information collection via devices

[0828] Specific explanation: The device is operated by a store clerk. First, the clerk launches the device's camera app and takes a picture of the "Today's Recommended Product." Then, he or she launches a voice recording app and records a description of the product and supplementary information about the limited-time sale.

[0829] Input: Product image, product description and audio information about the sale.

[0830] Output: Image data file, audio data file.

[0831] Specific operation: When a store clerk takes a photo of a product with the device, a "Photo" button appears on the screen. While recording audio, "Recording" is displayed, and a confirmation message is displayed after recording is complete.

[0832] Step 2:

[0833] Send data from the device to the server

[0834] Specifically: The device sends collected image and audio data to the server via Wi-Fi or mobile data network. The device application sends the collected data to the server's API endpoint.

[0835] Input: Image data files and audio data files stored on the device.

[0836] Output: Image data and audio data sent to the server.

[0837] Specific behavior: A progress bar will be displayed while the file is being sent, and once the sending is complete, a pop-up message will appear saying "Sending complete."

[0838] Step 3:

[0839] Data reception and analysis by the server

[0840] Specifically, the server receives image data and audio data sent from the device. It uses image recognition libraries such as OpenCV and TensorFlow to extract product names and prices from the received image data. It also converts the audio data into text using the Google Cloud Speech-to-Text API, from which it extracts the time of the limited-time sale and other important supplementary information.

[0841] Input: Image data and audio data sent to the server.

[0842] Output: Product information (product name, price), voice text information (limited sale time, supplementary information).

[0843] Specific operation: The progress of the analysis is recorded in the server log. For example, log information such as "Image recognition started," "Voice recognition started," and "Data analysis in progress" is displayed in real time.

[0844] Step 4:

[0845] Automatic flyer generation

[0846] Specifics: Automatically generate electronic flyers based on product information, prices, and time sale information collected by the server. Flyers are generated using the Adobe PDF library and Python's ReportLab library.

[0847] Input: Aggregated product information, pricing information, and sale information.

[0848] Output: e-flyer in PDF or image format.

[0849] Specific operation: When the flyer generation is completed, a notification "Flyer generation completed" will be sent to the administrator by email.

[0850] Step 5:

[0851] Electronic Delivery to Users

[0852] Detailed explanation: The server obtains the LINE account information of pre-registered users and distributes flyers generated using the LINE API. An image and PDF of the flyer are sent to the user's LINE account along with the message "Fresh tomatoes on sale today only!"

[0853] Input: Generated e-flyer, pre-registered user's LINE account information.

[0854] Output: A flyer sent to the user's LINE account.

[0855] Specific operation: The sending status of LINE messages is displayed in real time on the server dashboard. For example, the status is displayed as "Started sending," "Sent," or "Failed sending."

[0856] (Application example 1)

[0857] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0858] There is a need for a low-cost, effective method for each store in a shopping district to deliver discount information to consumers. In particular, there is currently no practical system that allows consumers to check discount information and product information in real time while walking around the store, and realizing this would be a challenge in improving the shopping district's overall ability to attract customers.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0860] In this invention, the server includes: a means for a terminal of a participating store to take an image of "Today's Recommended Product," a means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, a means for the server to receive image and audio data transmitted from the terminal, a means for the server to extract product information using image recognition, a means for the server to extract supplemental information from the audio data using voice recognition, a means for the server to aggregate the acquired information of all stores and automatically generate a flyer, a means for the server to electronically distribute the generated flyer to pre-registered users, a means for the smart glasses to display product information in the store in real time, a means for the smart glasses to provide additional information about products based on voice commands, and a means for the smart glasses to navigate to product shelves. This allows consumers to check the latest deals and product information in real time while walking around the store, increasing consumer purchasing motivation and improving the shopping district's overall customer attraction.

[0861] "Participating store terminals" are electronic devices owned and operated by each store in the shopping district, and have the function of taking pictures of "Today's Recommended Products" and recording audio descriptions of products and supplementary information about limited-time sales.

[0862] "Means for taking pictures of today's recommended products" refers to the function that allows store staff to take pictures of today's recommended products using a camera on a smartphone, tablet, or other device.

[0863] "Means for recording audio supplementary information about product descriptions and limited-time sales" refers to a function that allows store staff to use a microphone on a smartphone or tablet device to record audio supplementary information about product descriptions and limited-time sales.

[0864] "Server" refers to the central system that receives image data and audio data sent from the terminal, analyzes and processes this data, and generates discount information.

[0865] "Image recognition" refers to a technology that automatically extracts product information (such as product name and price) from image data received by the server.

[0866] "Speech recognition" refers to the technology that converts voice data received by the server into text and extracts important supplementary information from the text (such as time of sale and special information).

[0867] "Means for automatically generating flyers" refers to the function by which the server automatically creates electronic flyers containing information useful to users based on data obtained from image recognition and voice recognition.

[0868] "Means for electronic distribution to pre-registered users" refers to the function by which the server uses messaging services such as the LINE API to electronically distribute the generated flyer to the devices of pre-registered users.

[0869] "Smart glasses" refers to a wearable device that has a display that the wearer can see and that can display information in real time.

[0870] "Means for displaying product information within the store" refers to the function of the smart glasses to display information about the current location within the store and surrounding products on the display in real time.

[0871] "Means for providing additional information about a product based on a voice command" refers to the ability of the smart glasses to display more information about a product when the wearer requests it using a voice command.

[0872] "Means for navigating to product shelves" refers to the function of the smart glasses to guide the wearer to the desired product shelf based on their current location.

[0873] This invention is a system that enables each store in a shopping mall to distribute discount information to consumers at low cost, and provides product information in real time within the store using smart glasses. Specific embodiments for implementing the invention are described below.

[0874] System Configuration

[0875] The system consists of the following components:

[0876] 1. Store terminal: This refers to a smartphone or tablet device that store staff use to take photos of the "Today's Recommended Products" and record supplementary information via voice.

[0877] 2. Server: Receives image and audio data sent from the store's terminal, analyzes and processes it, and generates an electronic flyer.

[0878] 3. Smart glasses: A wearable device worn by the user that displays real-time store information and provides detailed product information based on voice commands.

[0879] Functional description of each element

[0880] 1. Device Function: The store clerk uses a smartphone or tablet to take pictures of the "Today's Recommended Products." They also use the voice recording function to record product descriptions and limited-time sale information. This data is first saved locally and then sent to the server.

[0881] 2. Server Functions:

[0882] The received image data is then subjected to image recognition using the Google Cloud Vision API, and product name and price information is automatically extracted.

[0883] The received audio data is converted into text using the Google Cloud Speech-to-Text API, from which the sale time and other important supplementary information are extracted.

[0884] Based on information collected from each store, electronic flyers are automatically generated in PDF or image format.

[0885] The generated flyers are distributed electronically to pre-registered users using the LINE API.

[0886] 3. Functions of smart glasses:

[0887] The smart glasses display real-time product information about the surrounding area as the user walks through the store.

[0888] Using voice commands, users can operate the smart glasses to obtain detailed product information.

[0889] The in-store navigation function provides directions to specific product shelves.

[0890] Specific examples

[0891] For example, if a user is looking for "fresh tomatoes," they can put on the smart glasses and walk around the store, which will do the following:

[0892] 1. Information collection: The camera in the smart glasses takes a picture of the tomato shelf, and the microphone records the salesperson saying, "These tomatoes are on sale. They're fresh and delicious."

[0893] 2. Information analysis: The server uses image recognition to extract the information "tomato" and "special price," and uses voice recognition to convert the information "special price" and "fresh" into text.

[0894] 3. Flyer generation: Based on the above information, an electronic flyer is created and distributed to users via LINE with the message "Fresh tomatoes on sale!"

[0895] Prompt Sentence Examples

[0896] An example of a prompt to input to a generative AI model is as follows:

[0897] ---

[0898] You are a smart eyeglasses assistant that provides real-time product information in the store. When the user approaches a specific product on the shelf, the display will show special offers and recommendations for that product.

[0899] Special Offers:

[0900] Product Name: Tomato

[0901] Price: 200 yen

[0902] Sale information: Special price campaign

[0903] Description: Fresh and delicious tomatoes.

[0904] Use this information to create an electronic flyer and distribute it to users.

[0905] ---

[0906] The above is a specific embodiment for carrying out the present invention. This system allows each store in a shopping mall to efficiently distribute advantageous information to consumers, and also allows consumers to obtain useful information in real time.

[0907] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0908] Step 1:

[0909] Information gathering

[0910] The device (smartphone or tablet) is operated by a store clerk. First, the clerk takes a picture of the "Today's Recommended Product." This is then saved locally as image data. At the same time, the device records the clerk's voice and saves the product description and supplementary information about the limited-time sale as audio data. This data is later sent to the server.

[0911] Input: Product images and audio recordings by store staff

[0912] Output: Image and audio data stored locally on the device

[0913] Step 2:

[0914] Data transmission

[0915] The device sends the image and audio data stored locally to the server via an internet connection.

[0916] Input: Locally stored image and audio data

[0917] Output: Image data and audio data sent to the server

[0918] Step 3:

[0919] Image Recognition

[0920] The server passes the received image data to the Google Cloud Vision API, which extracts product information from the product image. Specifically, it analyzes information such as the product name and price and obtains this as text data.

[0921] Input: Image data sent to the server

[0922] Output: Extracted product name and price text data

[0923] Step 4:

[0924] Voice Recognition

[0925] The server passes the received audio data to the Google Cloud Speech-to-Text API, converts the audio into text data, and then extracts the sale time and other supplemental information from the text data.

[0926] Input: Audio data sent to the server

[0927] Output: Text data of supplementary information extracted from the audio data

[0928] Step 5:

[0929] Information collection and flyer generation

[0930] The server aggregates the product information and supplementary information extracted in steps 3 and 4 to generate a single electronic flyer in PDF or image format.

[0931] Input: Text data of extracted product information and supplementary information

[0932] Output: Generated e-flyer (PDF or image format)

[0933] Step 6:

[0934] Electronic distribution

[0935] The server distributes the generated electronic flyers to pre-registered users using the LINE API, and users receive the flyers on their smartphones or other devices.

[0936] Input: Generated e-flyer

[0937] Output: Electronic flyer delivered to the user's device

[0938] Step 7:

[0939] Real-time information display

[0940] As users wear smart glasses and walk around the store, product information is displayed in real time on the glasses' display. This is achieved by the smart glasses knowing their current location within the store and receiving information from the server in real time.

[0941] Input: Real-time data from the server and the user's current location

[0942] Output: Product information displayed on the smart glasses display

[0943] Step 8:

[0944] Voice Commands and Navigation

[0945] When a user issues a voice command, it is recorded by the smart glasses' microphone and sent to the server. The server analyzes the voice command and sends detailed information about the corresponding product to the smart glasses. Navigation to specific product shelves is also based on the voice command.

[0946] Input: User voice command

[0947] Output: Product details and navigation information displayed on smart glasses

[0948] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0949] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[0950] 1. Information collection via devices

[0951] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The clerk shows the product to the terminal's camera, takes still and video images, and saves them as image files. Next, the clerk records audio into the terminal explaining the product and supplementary information about the limited-time sale, which is saved as an audio file.

[0952] 2. Information analysis by the server

[0953] The server receives the image and audio data sent from the device. For the received image data, it uses image recognition technology to extract product information (product name, price). For example, it recognizes text in the image and obtains necessary information from the product label and price display. For the received audio data, it uses voice recognition technology to convert the audio data into text and extracts the time of the limited sale and other special information.

[0954] 3. Generate flyers

[0955] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. The server automatically generates an electronic flyer based on this information, appropriately positioning the product name, price, and supplementary information to create an easy-to-understand flyer. This flyer is output in PDF or image format.

[0956] 4. Electronic Delivery

[0957] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can check the latest daily information on their smartphones, tablets, or other devices.

[0958] 5. User Emotion Recognition

[0959] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions from the user's facial expressions when viewing the flyer and from their voice comments about the flyer. It also collects feedback from the user after viewing the flyer and analyzes the emotion data collected.

[0960] 6. Information optimization

[0961] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer delivery. Specifically, it identifies the products and conditions that the user is interested in and adjusts the content of future flyers accordingly. This feedback loop makes it possible to provide information that matches the user's preferences.

[0962] Specific examples

[0963] For example, suppose a store inputs information into a device about a special sale on fresh tomatoes. In this case, the store clerk takes a photo of the tomatoes and their price information, and provides additional information in voice, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a flyer based on the extracted information and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction, allowing the content of future distributions to be optimized more effectively.

[0964] The above is a specific embodiment for carrying out the present invention. This invention allows consumers to easily obtain more advantageous product information, and each store in the shopping district can efficiently attract customers. Furthermore, by analyzing user reactions using an emotion engine, the information provided will constantly evolve to adapt to the user's preferences.

[0965] The processing flow will be explained below.

[0966] Step 1:

[0967] The device takes images and video of the "Today's Recommended Products." The store clerk points the device's camera at the product and takes a photo, saving it as an image file. Next, the store clerk records audio explaining the product and supplementary information about the limited-time sale, saving it as an audio file.

[0968] Step 2:

[0969] The device sends the captured image and audio files to the server. Specifically, the device sends the data to the server using an HTTP POST request. This request includes the store ID, image file, and audio file.

[0970] Step 3:

[0971] The server analyzes the image data received from the device. The server uses image recognition technology to extract product names, prices, and other relevant information. For example, the server recognizes text in the image and obtains the necessary information from product labels and price displays.

[0972] Step 4:

[0973] The server analyzes the voice data received from the device. The server converts the voice data into text using voice recognition technology, and extracts the time of the limited-time sale and special promotional information from the text. This information is recorded in conjunction with the product information.

[0974] Step 5:

[0975] The server aggregates the product information and supplementary information collected from all stores, compiles the data from each store in a unified format, and organizes it as information for the entire shopping district.

[0976] Step 6:

[0977] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create an easy-to-understand flyer. This flyer is output in PDF format or image format.

[0978] Step 7:

[0979] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, the server uses the LINE API to distribute it to each user. Users receive the flyer on their smartphones, tablets, or other devices.

[0980] Step 8:

[0981] The server uses an emotion engine to analyze the user's browsing behavior. Specifically, it collects facial expressions and voice comments while the user is browsing the flyer and analyzes the emotion data.

[0982] Step 9:

[0983] The server stores the user's emotional data recognized by the emotion engine, and refers to past emotional data to optimize the content of future flyer deliveries to suit the user's preferences.

[0984] The above steps complete the process in which the device collects information, the server analyzes and aggregates the information, generates flyers, and delivers them to users. Furthermore, an emotion engine is used to analyze user reactions, and the information provided constantly evolves to adapt to the user's preferences.

[0985] Example 2

[0986] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0987] There is a need for a system that allows each store in a shopping district to effectively distribute discount information to consumers at low cost, and also analyzes user reactions to optimize the content of the next distribution. However, currently, there are limited means to efficiently perform these tasks, and it is particularly difficult to analyze user reactions and reflect them in the next distribution. Therefore, in order to solve these problems, the present invention aims to provide a system that easily collects, distributes, and analyzes information, and enables continuous optimization.

[0988] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0989] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, means for the server to receive image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate flyers, means for the server to electronically distribute the generated flyers to pre-registered users, means for the server to analyze users' reactions while viewing the flyers, and means for the server to optimize the content of the next flyer based on the analysis results. This enables each store in the shopping district to distribute discount information efficiently at low cost and further analyze user reactions and reflect them in future flyers.

[0990] "Terminal" refers to a device operated by a staff member at a participating store, and is a device that has the function of recording and transmitting images and audio.

[0991] The "server" is a central system that receives collected image data and audio data, analyzes and processes them, and distributes the information.

[0992] "Image recognition" is a technology that allows a terminal or server to automatically extract specific information (for example, product name or price) from captured image data.

[0993] "Speech recognition" is a technology that analyzes voice data and extracts text data and information.

[0994] "Product information" refers to basic data related to a product, such as the product name and price.

[0995] "Supplementary information" is additional information that accompanies basic product information, such as time of sale or special information for a specific product.

[0996] A "flyer" is an advertising medium that is generated by aggregating product information and supplementary information collected from each store, and is output in PDF or image format.

[0997] "Electronic distribution" refers to the act of sending the generated flyer in digital form to pre-registered users.

[0998] The "emotion engine" is a system that analyzes emotions based on a user's facial expressions and voice comments, and recognizes specific emotional states.

[0999] "Optimization" is the process of adjusting the content of the next flyer based on the analysis results and providing information that matches the user's preferences and reactions.

[1000] "Reaction analysis" is a technology that analyzes the behavior and facial expressions of users when they view electronic flyers to determine their emotions and interests.

[1001] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[1002] Information collection via devices

[1003] The terminal is operated by a store clerk and can be a portable device such as a smartphone or tablet. The clerk uses the device's camera to take images or videos of "Today's Recommended Products" and saves them as image files. Next, the clerk uses the device's microphone to record audio descriptions of the products and supplementary information about the limited-time sale, which is saved as an audio file. For example, a clerk can take a picture of fresh tomatoes and say, "Today's limited-time sale, from 3:00 to 5:00 PM."

[1004] Server-based information analysis

[1005] The server receives the image and audio data sent from the device. It uses image recognition technology such as OpenCV or TensorFlow to extract product information (product name, price) from the received image data. For example, it analyzes the text in the image and obtains the necessary information from the product label and price display. It also converts the audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the time of the limited-time sale and other special information.

[1006] Generate flyers

[1007] The server aggregates product information and supplementary information collected from all stores. It compiles the data from each store in a unified format and organizes it as information for the entire shopping district. For example, it automatically generates electronic flyers based on the data from each store, appropriately positioning product names, prices, and sale information. The generated electronic flyers are output in PDF or image format.

[1008] Electronic distribution

[1009] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can easily check the latest daily information on their smartphones or tablets.

[1010] User Emotion Recognition

[1011] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions by analyzing the user's facial expressions and voice comments when they view the flyer. For example, if a user smiles when they view the flyer or leaves an interesting voice comment, that information is collected as emotion data.

[1012] Information Optimization

[1013] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer distribution. This allows the server to identify products and information that users are interested in and customize the content of the next flyer. For example, if a particular product received a good response in the previous distribution, the server can prioritize the distribution of information similar to that product.

[1014] Examples and prompts

[1015] For example, when a store inputs information that "fresh tomatoes are on sale at a special price," the store clerk takes a photo of the tomato and records the price information. Additionally, a voice message provides additional information, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a digital flyer based on the aggregated data and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction and optimizes the content of future deliveries.

[1016] An example of a prompt is as follows:

[1017] "We collect image and audio data of the 'Today's Recommended Products' provided by each store in the shopping district, and use image and audio recognition technology to extract product information and limited-time sale information. We then compile all of the information into a unified format to generate digital flyers and distribute them to users using the LINE API. We analyze users' emotional reactions and incorporate feedback into the next distribution."

[1018] The above is a specific embodiment for carrying out the present invention. This system enables each store in a shopping mall to efficiently distribute discount information and analyze user reactions to optimize the content of future distributions.

[1019] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1020] Step 1:

[1021] Information gathering and input data generation

[1022] Using the device's camera, the store clerk takes a picture of the "Today's Recommended Product" and saves it as an image file. Similarly, using the device's microphone, the store clerk records audio of the product description and supplementary information about the limited-time sale, and saves it as an audio file. In actual operation, for example, a store clerk can take a picture of a fresh tomato and explain in audio, "Today's limited-time sale, from 3:00 PM to 5:00 PM." The input for this step is the product image and audio of the product description, and the output is an image file and an audio file.

[1023] Step 2:

[1024] Sending and receiving data to the server

[1025] The device sends the generated image and audio files to the server. The server receives these data and prepares them for analysis. The input in this step is the data sent from the device (image and audio files), and the output is these files stored on the server.

[1026] Step 3:

[1027] Processing image data and extracting product information

[1028] The server uses image recognition technology to extract product information (product name, price) from the received image file. For example, it uses tools such as OpenCV or TensorFlow to analyze the text in the image and obtain the necessary information from the product label and price display. The input for this process is the image file, and the output is the extracted product name and price information.

[1029] Step 4:

[1030] Processing audio data and extracting supplementary information

[1031] The server converts the received audio file into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the sale time and other special information. The input in this step is the audio file, and the output is the text supplementary information.

[1032] Step 5:

[1033] Generate electronic flyers

[1034] The server aggregates the product information and supplementary information collected from all stores and automatically generates an electronic flyer. Specifically, it organizes the collected data into a unified format and arranges the product name, price, and supplementary information in an appropriate layout. The input in this step is the product information and supplementary information collected from each store, and the output is an electronic flyer in PDF or image format.

[1035] Step 6:

[1036] Electronic distribution

[1037] The server sends the generated e-flyer to the user's pre-registered LINE account. It distributes it to the user's smartphone or tablet using the LINE API. The input in this step is the generated e-flyer, and the output is the e-flyer distributed to the user's device.

[1038] Step 7:

[1039] User sentiment analysis

[1040] The server uses an emotion engine to analyze the user's reactions while they are viewing the electronic flyer. Specifically, it analyzes the user's facial expressions and voice comments to obtain emotional information. The input in this step is the user's viewing data (facial expressions, voice comments), and the output is the analyzed emotional data.

[1041] Step 8:

[1042] Information Optimization

[1043] The server saves the emotion data obtained by the emotion engine and reflects it in future flyer content. It analyzes past data to identify products and information that the user was particularly interested in, and customizes the content of the next flyer based on that. The input in this step is the analyzed emotion data, and the output is an optimized version of the next digital flyer.

[1044] The above is the specific flow of the program processing of this system.

[1045] (Application example 2)

[1046] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] Traditional shopping districts and retail stores lacked efficient methods for advertising discount information and seasonal products, resulting in insufficient communication of information to consumers and challenges in promoting store visits and sales. Furthermore, there was a lack of methods for collecting and analyzing consumer responses in real time and reflecting them in the next marketing strategy, making it difficult to respond quickly to consumer needs. This made it difficult to effectively attract customers amid intensifying competition between stores.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1049] In this invention, the server includes a means for the terminal to capture and record image and audio data, a means for analyzing the data received by the server, a means for analyzing user reactions using an emotion engine, and a means for optimizing the content of the next flyer. This enables shopping malls and retail stores to efficiently distribute information to consumers and analyze their reactions, enabling them to respond quickly to consumer needs.

[1050] "Participating store terminal" refers to equipment used by each participating store to take product images and record audio.

[1051] "Today's Recommended Products" are products that are especially recommended on that day, and refer to products or services that the store wants to promote in a special way.

[1052] "Means for taking images" refers to the function of taking images of products using a device with a camera function and saving them as data.

[1053] "Means of recording audio" refers to the ability to record product descriptions and supplementary information about limited-time sales using a device with a microphone function and save it as audio data.

[1054] A "server" is a central processing unit that analyzes image and audio data and performs various processes.

[1055] The "means for receiving" refers to the function of receiving data transmitted from a terminal.

[1056] "Image recognition" refers to the technology of automatically analyzing and extracting characters and objects from image data.

[1057] "Product information" refers to data that indicates the product name, price, and other related information.

[1058] "Speech recognition" refers to the technology of converting voice data into text data and analyzing it.

[1059] "Additional Information" refers to sale times and other special information.

[1060] "Means for automatically generating flyers" refers to a function that automatically generates visually appealing flyers based on collected data.

[1061] "Means of electronic distribution" refers to the ability to distribute the generated flyer via email or messaging apps.

[1062] "User" refers to a consumer who wishes to receive electronic flyers and has registered in advance.

[1063] "Emotion engine" refers to technology that analyzes emotions from users' facial expressions and feedback comments.

[1064] "Means for analyzing reactions" refers to a function that uses an emotion engine to analyze the reactions of users when they view a flyer.

[1065] "Optimization means" refers to the function of adjusting the content of future flyers based on analyzed response data, and providing optimal information tailored to consumer preferences.

[1066] The present invention is a system for enabling shopping malls and retail stores to efficiently distribute discount information to consumers and analyze their reactions. The system includes a terminal, a server, and an emotion engine for analyzing user reactions.

[1067] 1. Information collection via devices

[1068] The terminal is a device operated by a store clerk at a participating store, and can be a smartphone or tablet. The clerk uses the device's camera to take pictures of the "Today's Recommended Products" and also records audio descriptions of the products and supplementary information about the limited-time sale. This data is temporarily stored on the terminal and later sent to the server.

[1069] 2. Information analysis by the server

[1070] The server receives the image and audio data sent from the device and performs the main analysis process. The image data is converted into text using image recognition software (e.g., Amazon Rekognition) to extract product name and price information. Similarly, the audio data is converted into text using speech recognition software (e.g., Google Cloud Speech-to-Text) to extract limited-time sale times and other special information.

[1071] 3. Generate flyers

[1072] The server aggregates product information and supplemental information collected from all stores and automatically generates visually appealing digital flyers using template generation tools such as Adobe InDesign and Canva API. The generated flyers are saved in PDF format.

[1073] 4. Electronic Delivery

[1074] The server then distributes the generated digital flyers to the LINE accounts of pre-registered users, using the LINE Messaging API, allowing users to check the latest information on their smartphones, tablets, or other devices.

[1075] 5. User Emotion Recognition

[1076] The server uses an emotion engine (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer) to analyze the user's reaction when viewing the flyer. It analyzes the user's facial expressions and feedback comments to recognize positive, negative, and neutral emotions.

[1077] 6. Information optimization

[1078] The server then optimizes the content of the next flyer based on the analyzed emotion data. This allows the display of product information that the user is interested in to be prioritized, improving advertising effectiveness. For example, if the user shows interest in fresh tomatoes, the next flyer will prominently feature information about fresh vegetables and special offers.

[1079] Specific examples

[1080] For example, if a store is selling fresh tomatoes at a special price, a store clerk can take a picture of the tomato with their smartphone and record a voice description of the product, saying, "Today only, special price on sale from 3:00 PM to 5:00 PM." This data is sent to a server and analyzed. The generated flyer is then sent to users via LINE, and the emotion engine analyzes their reactions. As a result, the content of the next flyer can be optimized more effectively.

[1081] Prompt Sentence Examples

[1082] Image analysis prompt:

[1083] Image file name: store_product.jpg

[1084] API used: Image recognition software

[1085] Information to be recognized: Product name, price

[1086] Prompt for speech analysis:

[1087] Audio file name: product_description.wav

[1088] API used: Speech recognition software

[1089] Extracted information: Time of sale, special information

[1090] Emotion recognition prompt:

[1091] User feedback: "These tomatoes look fresh and delicious."

[1092] API used: Emotion recognition software

[1093] Emotions to extract: positive, negative, neutral

[1094] By combining these technologies, shopping districts and retail stores can provide consumers with discount information at the optimal time, enabling effective marketing.

[1095] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1096] Step 1:

[1097] The device takes a picture of the "Today's Recommended Product." The input is a product image taken using the device's camera function, and the output is saved as an image file in the device's storage. A store clerk takes a picture of the product using the device's camera and saves the image.

[1098] Step 2:

[1099] The device records audio for product descriptions and supplementary information about limited-time sales. The input is audio recorded through a microphone, and the output is saved as an audio file in the device's storage. A store clerk explains the product into the microphone, and the audio is saved.

[1100] Step 3:

[1101] The device sends the captured images and recorded audio data to the server. The input is the image and audio files captured and recorded on the device, and the output is obtained by transferring this data to the server. The device uploads the data to the server via the network.

[1102] Step 4:

[1103] The server analyzes the image data it receives. The input is the image file received by the server, and the output is product information extracted from the text and objects in the image. The server analyzes the image using image recognition software (e.g., Amazon Rekognition) and extracts product name and price information.

[1104] Step 5:

[1105] The server analyzes the received audio data. The input is the audio file received by the server, and the output is supplementary information converted from the audio into text. The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text and analyze it.

[1106] Step 6:

[1107] The server aggregates product information and supplementary information collected from all stores and automatically generates flyers. The input is the analyzed product information and supplementary information, and the output is an electronic flyer. The server uses a template generation tool (e.g., Adobe InDesign, Canva API) to generate flyers based on the acquired data.

[1108] Step 7:

[1109] The server distributes the generated e-flyer to pre-registered users. The input is the generated e-flyer and the user's contact information, and the output is the flyer distributed to the recipient user. The server uses the LINE Messaging API to send the flyer to the user's LINE account.

[1110] Step 8:

[1111] The server analyzes the user's reaction when viewing the flyer. The input is the user's facial expression data and feedback comments, and the output is the user's emotion recognition results. The server analyzes the user's reaction using emotion recognition software (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer).

[1112] Step 9:

[1113] The server optimizes the content of the next flyer based on the analysis results. The input is the analyzed emotion data, and the output is the optimized flyer content. Based on the feedback obtained, the server prioritizes product information that matches the user's preferences and reflects it in the next flyer.

[1114] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1115] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1116] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1117] [Fourth embodiment]

[1118] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1119] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1120] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1121] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1122] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1123] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1124] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1125] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1126] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1127] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1128] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1129] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1130] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1131] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[1132] 1. Information collection via devices

[1133] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The captured images clearly display the products, making them attractive and easy to understand for consumers. The terminal also has a voice recording function, allowing the clerk to record a voice explanation of the products and supplementary information about the limited-time sale. The terminal stores this data locally and then transmits it to the server.

[1134] 2. Information analysis by the server

[1135] The server receives the image and audio data sent from the terminal. It uses image recognition technology to extract product information (such as product name and price) from the received image data. The server also uses voice recognition technology to convert the audio data into text, from which it extracts the time of limited-time sales and other important supplementary information. This makes it possible to integrate and consolidate information for the entire shopping district, rather than just for individual stores.

[1136] 3. Generate flyers

[1137] The server automatically generates electronic flyers based on the information collected from each store. These flyers contain product names, prices, and time sale information, and are laid out so that users can easily see all the useful information at a glance. These flyers are output as electronic documents in PDF or image format.

[1138] 4. Electronic Delivery

[1139] The server obtains the LINE account information of pre-registered users. Once the flyer is generated, it is distributed using the LINE API. Users can receive daily updated discount information on their smartphones, tablets, and other devices. This allows stores in shopping districts to deliver the latest information directly to consumers without having to use large-scale advertising methods.

[1140] Specific examples

[1141] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[1142] The above is a specific embodiment for carrying out the present invention. The present invention allows consumers to easily obtain more advantageous product information, and each store in the shopping mall can efficiently attract customers.

[1143] The processing flow will be explained below.

[1144] Step 1:

[1145] The device takes images and videos of "Today's Recommended Products." The store clerk takes still and video images of the products shown to the device's camera, which are saved as image files. Next, the store clerk records audio into the device explaining the products and supplementary information about the limited-time sale, which is saved as an audio file.

[1146] Step 2:

[1147] The device sends the captured image and audio files to the server. Specifically, the data is sent using an HTTP POST request. This request includes the store ID, image file, and audio file, and the server temporarily stores them.

[1148] Step 3:

[1149] The server analyzes the image data received from the device and uses image recognition technology to extract product names, prices, and other relevant information, for example by recognizing text in the image and obtaining the necessary information from product labels and price displays.

[1150] Step 4:

[1151] The server analyzes the voice data received from the device. It uses voice recognition technology to convert the voice data into text. From the converted text, it extracts supplemental information such as time of sale and special promotional content. This information is recorded in conjunction with the product information.

[1152] Step 5:

[1153] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. This allows information from multiple stores to be aggregated into a single flyer.

[1154] Step 6:

[1155] The server automatically generates an electronic flyer based on the aggregated information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create a visually easy-to-understand flyer. This flyer is output as a PDF or image file.

[1156] Step 7:

[1157] The server obtains the LINE account information of pre-registered users. It then distributes flyers generated using the LINE API to each user. Users can receive the flyers on their smartphones, tablets, or other devices and check the latest daily information.

[1158] The above steps complete a series of processes in which the terminal collects information, the server analyzes and aggregates the information, generates a flyer, and distributes it to the user.

[1159] Example 1

[1160] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1161] Currently, there is a lack of means for each store in a shopping district to efficiently distribute discount information to consumers at low cost. Conventional advertising methods require large amounts of advertising expenses, which places a heavy burden on small stores in shopping districts. In addition, there are limited means of providing consumers with the latest information in a timely manner. This creates challenges for the shopping district as a whole in attracting customers and communicating with consumers.

[1162] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1163] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio for product descriptions and supplemental information about limited-time sales, means for the terminal to transmit image data and audio data collected by the terminal to the server, means for the server to receive the image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate a flyer in electronic document format, and means for electronically distributing the flyer generated by the server to pre-registered users. This enables each store in the shopping district to provide consumers with the latest information effectively and at low cost, thereby increasing its customer attraction.

[1164] A "participating store terminal" is a device with image capture and audio recording functions that can be operated by store staff at each store in the shopping district.

[1165] "Image of today's recommended product" is a photograph or image file that shows the visual information of the product that each store in the shopping district wants to particularly promote that day.

[1166] "Supplementary information on product descriptions and limited-time sales" refers to audio recordings of detailed product descriptions and sales information.

[1167] "Image data" refers to image files of products photographed by each store in the shopping mall using a terminal.

[1168] "Audio data" refers to audio files containing product descriptions and sales information recorded by each store in the shopping mall using a terminal.

[1169] A "server" is a central computer system that receives data sent from terminals and performs analysis and information aggregation.

[1170] "Image recognition" is a technology used by the server to extract product name and price information from image data.

[1171] "Speech recognition" is a technology used by the server to convert voice data into text data and extract the sale times and other supplemental information from it.

[1172] "Flyers in electronic document format" are advertising materials that can be saved and displayed as electronic data, such as PDF or image formats.

[1173] A "pre-registered user" is a consumer who has registered in advance with the system to receive information about the shopping mall.

[1174] "Electronic distribution" is the means of sending flyers and other information to users' devices over the internet.

[1175] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost. Specific embodiments for carrying out the present invention will be described below.

[1176] Information collection via devices

[1177] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." For example, the terminal is equipped with a high-resolution camera and voice recording function, allowing a store clerk to take a picture of a fresh tomato and record a voice explanation such as "This tomato is on sale for today only." This terminal can be used with a regular smartphone or tablet. The terminal's application has also been developed as software specifically for the store, making it easy to take pictures and record voice. The terminal saves this data locally and then sends it to the server.

[1178] Send data from the device to the server

[1179] The device application sends the stored image and audio data to the server via Wi-Fi or a mobile data network. Specifically, the device application sends the collected data to the server's API endpoint, and a confirmation message is displayed on the device indicating that the data has been sent.

[1180] Data reception and analysis by the server

[1181] The server receives image data and audio data sent from the device. The received image data is analyzed using image recognition libraries such as OpenCV and TensorFlow to extract product names and prices. For example, from an image of fresh tomatoes, the product name "tomato" and the photographed price information can be recognized. The audio data is also converted into text using the Google Cloud Speech-to-Text API, allowing the extraction of time-limited sale dates and other important supplementary information. The progress of the analysis is recorded in the server log, making it possible to check in real time.

[1182] Automatic flyer generation

[1183] The server automatically generates electronic flyers based on the information collected from each store. For example, using the Adobe PDF library or Python's ReportLab library, flyers containing collected product information, prices, and time sale information are generated. The flyers are saved in PDF or image format, and a notification of generation completion is sent to the administrator by email.

[1184] Electronic Delivery to Users

[1185] The server obtains the LINE account information of pre-registered users and distributes the flyer using the LINE API once it has been generated. For example, the server sends an image or PDF of the flyer to the user's LINE account along with the message "Fresh tomatoes on sale today only!" The status of the distribution is displayed in real time on the server's dashboard, allowing users to check the status.

[1186] Specific examples

[1187] For example, a store might input information about a special sale on fresh tomatoes into a terminal. In this case, the store clerk takes a picture of the tomato and provides a voice explanation of the price and sale period. The terminal then sends this information to a server, which then extracts the information using image and voice recognition. As a result, the server generates a flyer containing an image of the tomato and the special sale information, and distributes it to registered users. This process allows users to plan their shopping trips at the mall while comparing it with the supermarket flyer.

[1188] Example prompts for generative AI models

[1189] "There's a special price on fresh tomatoes at the mall. How can you let users know about this?"

[1190] The above is a specific embodiment for carrying out the present invention. This system allows consumers to easily obtain information on more advantageous products, and each store in the shopping mall can efficiently attract customers.

[1191] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1192] Step 1:

[1193] Information collection via devices

[1194] Specific explanation: The device is operated by a store clerk. First, the clerk launches the device's camera app and takes a picture of the "Today's Recommended Product." Then, he or she launches a voice recording app and records a description of the product and supplementary information about the limited-time sale.

[1195] Input: Product image, product description and audio information about the sale.

[1196] Output: Image data file, audio data file.

[1197] Specific operation: When a store clerk takes a photo of a product with the device, a "Photo" button appears on the screen. While recording audio, "Recording" is displayed, and a confirmation message is displayed after recording is complete.

[1198] Step 2:

[1199] Send data from the device to the server

[1200] Specifically: The device sends collected image and audio data to the server via Wi-Fi or mobile data network. The device application sends the collected data to the server's API endpoint.

[1201] Input: Image data files and audio data files stored on the device.

[1202] Output: Image data and audio data sent to the server.

[1203] Specific behavior: A progress bar will be displayed while the file is being sent, and once the sending is complete, a pop-up message will appear saying "Sending complete."

[1204] Step 3:

[1205] Data reception and analysis by the server

[1206] Specifically, the server receives image data and audio data sent from the device. It uses image recognition libraries such as OpenCV and TensorFlow to extract product names and prices from the received image data. It also converts the audio data into text using the Google Cloud Speech-to-Text API, from which it extracts the time of the limited-time sale and other important supplementary information.

[1207] Input: Image data and audio data sent to the server.

[1208] Output: Product information (product name, price), voice text information (limited sale time, supplementary information).

[1209] Specific operation: The progress of the analysis is recorded in the server log. For example, log information such as "Image recognition started," "Voice recognition started," and "Data analysis in progress" is displayed in real time.

[1210] Step 4:

[1211] Automatic flyer generation

[1212] Specifics: Automatically generate electronic flyers based on product information, prices, and time sale information collected by the server. Flyers are generated using the Adobe PDF library and Python's ReportLab library.

[1213] Input: Aggregated product information, pricing information, and sale information.

[1214] Output: e-flyer in PDF or image format.

[1215] Specific operation: When the flyer generation is completed, a notification "Flyer generation completed" will be sent to the administrator by email.

[1216] Step 5:

[1217] Electronic Delivery to Users

[1218] Detailed explanation: The server obtains the LINE account information of pre-registered users and distributes flyers generated using the LINE API. An image and PDF of the flyer are sent to the user's LINE account along with the message "Fresh tomatoes on sale today only!"

[1219] Input: Generated e-flyer, pre-registered user's LINE account information.

[1220] Output: A flyer sent to the user's LINE account.

[1221] Specific operation: The sending status of LINE messages is displayed in real time on the server dashboard. For example, the status is displayed as "Started sending," "Sent," or "Failed sending."

[1222] (Application example 1)

[1223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] There is a need for a low-cost, effective method for each store in a shopping district to deliver discount information to consumers. In particular, there is currently no practical system that allows consumers to check discount information and product information in real time while walking around the store, and realizing this would be a challenge in improving the shopping district's overall ability to attract customers.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1226] In this invention, the server includes: a means for a terminal of a participating store to take an image of "Today's Recommended Product," a means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, a means for the server to receive image and audio data transmitted from the terminal, a means for the server to extract product information using image recognition, a means for the server to extract supplemental information from the audio data using voice recognition, a means for the server to aggregate the acquired information of all stores and automatically generate a flyer, a means for the server to electronically distribute the generated flyer to pre-registered users, a means for the smart glasses to display product information in the store in real time, a means for the smart glasses to provide additional information about products based on voice commands, and a means for the smart glasses to navigate to product shelves. This allows consumers to check the latest deals and product information in real time while walking around the store, increasing consumer purchasing motivation and improving the shopping district's overall customer attraction.

[1227] "Participating store terminals" are electronic devices owned and operated by each store in the shopping district, and have the function of taking pictures of "Today's Recommended Products" and recording audio descriptions of products and supplementary information about limited-time sales.

[1228] "Means for taking pictures of today's recommended products" refers to the function that allows store staff to take pictures of today's recommended products using a camera on a smartphone, tablet, or other device.

[1229] "Means for recording audio supplementary information about product descriptions and limited-time sales" refers to a function that allows store staff to use a microphone on a smartphone or tablet device to record audio supplementary information about product descriptions and limited-time sales.

[1230] "Server" refers to the central system that receives image data and audio data sent from the terminal, analyzes and processes this data, and generates discount information.

[1231] "Image recognition" refers to a technology that automatically extracts product information (such as product name and price) from image data received by the server.

[1232] "Speech recognition" refers to the technology that converts voice data received by the server into text and extracts important supplementary information from the text (such as time of sale and special information).

[1233] "Means for automatically generating flyers" refers to the function by which the server automatically creates electronic flyers containing information useful to users based on data obtained from image recognition and voice recognition.

[1234] "Means for electronic distribution to pre-registered users" refers to the function by which the server uses messaging services such as the LINE API to electronically distribute the generated flyer to the devices of pre-registered users.

[1235] "Smart glasses" refers to a wearable device that has a display that the wearer can see and that can display information in real time.

[1236] "Means for displaying product information within the store" refers to the function of the smart glasses to display information about the current location within the store and surrounding products on the display in real time.

[1237] "Means for providing additional information about a product based on a voice command" refers to the ability of the smart glasses to display more information about a product when the wearer requests it using a voice command.

[1238] "Means for navigating to product shelves" refers to the function of the smart glasses to guide the wearer to the desired product shelf based on their current location.

[1239] This invention is a system that enables each store in a shopping mall to distribute discount information to consumers at low cost, and provides product information in real time within the store using smart glasses. Specific embodiments for implementing the invention are described below.

[1240] System Configuration

[1241] The system consists of the following components:

[1242] 1. Store terminal: This refers to a smartphone or tablet device that store staff use to take photos of the "Today's Recommended Products" and record supplementary information via voice.

[1243] 2. Server: Receives image and audio data sent from the store's terminal, analyzes and processes it, and generates an electronic flyer.

[1244] 3. Smart glasses: A wearable device worn by the user that displays real-time store information and provides detailed product information based on voice commands.

[1245] Functional description of each element

[1246] 1. Device Function: The store clerk uses a smartphone or tablet to take pictures of the "Today's Recommended Products." They also use the voice recording function to record product descriptions and limited-time sale information. This data is first saved locally and then sent to the server.

[1247] 2. Server Functions:

[1248] The received image data is then subjected to image recognition using the Google Cloud Vision API, and product name and price information is automatically extracted.

[1249] The received audio data is converted into text using the Google Cloud Speech-to-Text API, from which the sale time and other important supplementary information are extracted.

[1250] Based on information collected from each store, electronic flyers are automatically generated in PDF or image format.

[1251] The generated flyers are distributed electronically to pre-registered users using the LINE API.

[1252] 3. Functions of smart glasses:

[1253] The smart glasses display real-time product information about the surrounding area as the user walks through the store.

[1254] Using voice commands, users can operate the smart glasses to obtain detailed product information.

[1255] The in-store navigation function provides directions to specific product shelves.

[1256] Specific examples

[1257] For example, if a user is looking for "fresh tomatoes," they can put on the smart glasses and walk around the store, which will do the following:

[1258] 1. Information collection: The camera in the smart glasses takes a picture of the tomato shelf, and the microphone records the salesperson saying, "These tomatoes are on sale. They're fresh and delicious."

[1259] 2. Information analysis: The server uses image recognition to extract the information "tomato" and "special price," and uses voice recognition to convert the information "special price" and "fresh" into text.

[1260] 3. Flyer generation: Based on the above information, an electronic flyer is created and distributed to users via LINE with the message "Fresh tomatoes on sale!"

[1261] Prompt Sentence Examples

[1262] An example of a prompt to input to a generative AI model is as follows:

[1263] ---

[1264] You are a smart eyeglasses assistant that provides real-time product information in the store. When the user approaches a specific product on the shelf, the display will show special offers and recommendations for that product.

[1265] Special Offers:

[1266] Product Name: Tomato

[1267] Price: 200 yen

[1268] Sale information: Special price campaign

[1269] Description: Fresh and delicious tomatoes.

[1270] Use this information to create an electronic flyer and distribute it to users.

[1271] ---

[1272] The above is a specific embodiment for carrying out the present invention. This system allows each store in a shopping mall to efficiently distribute advantageous information to consumers, and also allows consumers to obtain useful information in real time.

[1273] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1274] Step 1:

[1275] Information gathering

[1276] The device (smartphone or tablet) is operated by a store clerk. First, the clerk takes a picture of the "Today's Recommended Product." This is then saved locally as image data. At the same time, the device records the clerk's voice and saves the product description and supplementary information about the limited-time sale as audio data. This data is later sent to the server.

[1277] Input: Product images and audio recordings by store staff

[1278] Output: Image and audio data stored locally on the device

[1279] Step 2:

[1280] Data transmission

[1281] The device sends the image and audio data stored locally to the server via an internet connection.

[1282] Input: Locally stored image and audio data

[1283] Output: Image data and audio data sent to the server

[1284] Step 3:

[1285] Image Recognition

[1286] The server passes the received image data to the Google Cloud Vision API, which extracts product information from the product image. Specifically, it analyzes information such as the product name and price and obtains this as text data.

[1287] Input: Image data sent to the server

[1288] Output: Extracted product name and price text data

[1289] Step 4:

[1290] Voice Recognition

[1291] The server passes the received audio data to the Google Cloud Speech-to-Text API, converts the audio into text data, and then extracts the sale time and other supplemental information from the text data.

[1292] Input: Audio data sent to the server

[1293] Output: Text data of supplementary information extracted from the audio data

[1294] Step 5:

[1295] Information collection and flyer generation

[1296] The server aggregates the product information and supplementary information extracted in steps 3 and 4 to generate a single electronic flyer in PDF or image format.

[1297] Input: Text data of extracted product information and supplementary information

[1298] Output: Generated e-flyer (PDF or image format)

[1299] Step 6:

[1300] Electronic distribution

[1301] The server distributes the generated electronic flyers to pre-registered users using the LINE API, and users receive the flyers on their smartphones or other devices.

[1302] Input: Generated e-flyer

[1303] Output: Electronic flyer delivered to the user's device

[1304] Step 7:

[1305] Real-time information display

[1306] As users wear smart glasses and walk around the store, product information is displayed in real time on the glasses' display. This is achieved by the smart glasses knowing their current location within the store and receiving information from the server in real time.

[1307] Input: Real-time data from the server and the user's current location

[1308] Output: Product information displayed on the smart glasses display

[1309] Step 8:

[1310] Voice Commands and Navigation

[1311] When a user issues a voice command, it is recorded by the smart glasses' microphone and sent to the server. The server analyzes the voice command and sends detailed information about the corresponding product to the smart glasses. Navigation to specific product shelves is also based on the voice command.

[1312] Input: User voice command

[1313] Output: Product details and navigation information displayed on smart glasses

[1314] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1315] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[1316] 1. Information collection via devices

[1317] The terminal is operated by a store clerk and has the function of taking pictures of "Today's Recommended Products." The clerk shows the product to the terminal's camera, takes still and video images, and saves them as image files. Next, the clerk records audio into the terminal explaining the product and supplementary information about the limited-time sale, which is saved as an audio file.

[1318] 2. Information analysis by the server

[1319] The server receives the image and audio data sent from the device. For the received image data, it uses image recognition technology to extract product information (product name, price). For example, it recognizes text in the image and obtains necessary information from the product label and price display. For the received audio data, it uses voice recognition technology to convert the audio data into text and extracts the time of the limited sale and other special information.

[1320] 3. Generate flyers

[1321] The server aggregates product information and supplementary information collected from all stores. The data from each store is compiled in a unified format and organized as information for the entire shopping district. The server automatically generates an electronic flyer based on this information, appropriately positioning the product name, price, and supplementary information to create an easy-to-understand flyer. This flyer is output in PDF or image format.

[1322] 4. Electronic Delivery

[1323] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can check the latest daily information on their smartphones, tablets, or other devices.

[1324] 5. User Emotion Recognition

[1325] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions from the user's facial expressions when viewing the flyer and from their voice comments about the flyer. It also collects feedback from the user after viewing the flyer and analyzes the emotion data collected.

[1326] 6. Information optimization

[1327] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer delivery. Specifically, it identifies the products and conditions that the user is interested in and adjusts the content of future flyers accordingly. This feedback loop makes it possible to provide information that matches the user's preferences.

[1328] Specific examples

[1329] For example, suppose a store inputs information into a device about a special sale on fresh tomatoes. In this case, the store clerk takes a photo of the tomatoes and their price information, and provides additional information in voice, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a flyer based on the extracted information and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction, allowing the content of future distributions to be optimized more effectively.

[1330] The above is a specific embodiment for carrying out the present invention. This invention allows consumers to easily obtain more advantageous product information, and each store in the shopping district can efficiently attract customers. Furthermore, by analyzing user reactions using an emotion engine, the information provided will constantly evolve to adapt to the user's preferences.

[1331] The processing flow will be explained below.

[1332] Step 1:

[1333] The device takes images and video of the "Today's Recommended Products." The store clerk points the device's camera at the product and takes a photo, saving it as an image file. Next, the store clerk records audio explaining the product and supplementary information about the limited-time sale, saving it as an audio file.

[1334] Step 2:

[1335] The device sends the captured image and audio files to the server. Specifically, the device sends the data to the server using an HTTP POST request. This request includes the store ID, image file, and audio file.

[1336] Step 3:

[1337] The server analyzes the image data received from the device. The server uses image recognition technology to extract product names, prices, and other relevant information. For example, the server recognizes text in the image and obtains the necessary information from product labels and price displays.

[1338] Step 4:

[1339] The server analyzes the voice data received from the device. The server converts the voice data into text using voice recognition technology, and extracts the time of the limited-time sale and special promotional information from the text. This information is recorded in conjunction with the product information.

[1340] Step 5:

[1341] The server aggregates the product information and supplementary information collected from all stores, compiles the data from each store in a unified format, and organizes it as information for the entire shopping district.

[1342] Step 6:

[1343] The server automatically generates an electronic flyer based on the collected information. Using a PDF generation library, the product name, price, and supplementary information are properly arranged to create an easy-to-understand flyer. This flyer is output in PDF format or image format.

[1344] Step 7:

[1345] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, the server uses the LINE API to distribute it to each user. Users receive the flyer on their smartphones, tablets, or other devices.

[1346] Step 8:

[1347] The server uses an emotion engine to analyze the user's browsing behavior. Specifically, it collects facial expressions and voice comments while the user is browsing the flyer and analyzes the emotion data.

[1348] Step 9:

[1349] The server stores the user's emotional data recognized by the emotion engine, and refers to past emotional data to optimize the content of future flyer deliveries to suit the user's preferences.

[1350] The above steps complete the process in which the device collects information, the server analyzes and aggregates the information, generates flyers, and delivers them to users. Furthermore, an emotion engine is used to analyze user reactions, and the information provided constantly evolves to adapt to the user's preferences.

[1351] Example 2

[1352] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1353] There is a need for a system that allows each store in a shopping district to effectively distribute discount information to consumers at low cost, and also analyzes user reactions to optimize the content of the next distribution. However, currently, there are limited means to efficiently perform these tasks, and it is particularly difficult to analyze user reactions and reflect them in the next distribution. Therefore, in order to solve these problems, the present invention aims to provide a system that easily collects, distributes, and analyzes information, and enables continuous optimization.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1355] In this invention, the server includes means for a terminal of a participating store to take an image of "Today's Recommended Product," means for the terminal to record audio of product descriptions and supplemental information about limited-time sales, means for the server to receive image and audio data transmitted from the terminal, means for the server to extract product information using image recognition, means for the server to extract supplemental information from the audio data using audio recognition, means for the server to aggregate the information of all stores acquired by the server and automatically generate flyers, means for the server to electronically distribute the generated flyers to pre-registered users, means for the server to analyze users' reactions while viewing the flyers, and means for the server to optimize the content of the next flyer based on the analysis results. This enables each store in the shopping district to distribute discount information efficiently at low cost and further analyze user reactions and reflect them in future flyers.

[1356] "Terminal" refers to a device operated by a staff member at a participating store, and is a device that has the function of recording and transmitting images and audio.

[1357] The "server" is a central system that receives collected image data and audio data, analyzes and processes them, and distributes the information.

[1358] "Image recognition" is a technology that allows a terminal or server to automatically extract specific information (for example, product name or price) from captured image data.

[1359] "Speech recognition" is a technology that analyzes voice data and extracts text data and information.

[1360] "Product information" refers to basic data related to a product, such as the product name and price.

[1361] "Supplementary information" is additional information that accompanies basic product information, such as time of sale or special information for a specific product.

[1362] A "flyer" is an advertising medium that is generated by aggregating product information and supplementary information collected from each store, and is output in PDF or image format.

[1363] "Electronic distribution" refers to the act of sending the generated flyer in digital form to pre-registered users.

[1364] The "emotion engine" is a system that analyzes emotions based on a user's facial expressions and voice comments, and recognizes specific emotional states.

[1365] "Optimization" is the process of adjusting the content of the next flyer based on the analysis results and providing information that matches the user's preferences and reactions.

[1366] "Reaction analysis" is a technology that analyzes the behavior and facial expressions of users when they view electronic flyers to determine their emotions and interests.

[1367] The present invention relates to a system for enabling each store in a shopping mall to distribute bargain information to consumers at low cost and for analyzing user responses. Specific embodiments for carrying out the present invention will be described below.

[1368] Information collection via devices

[1369] The terminal is operated by a store clerk and can be a portable device such as a smartphone or tablet. The clerk uses the device's camera to take images or videos of "Today's Recommended Products" and saves them as image files. Next, the clerk uses the device's microphone to record audio descriptions of the products and supplementary information about the limited-time sale, which is saved as an audio file. For example, a clerk can take a picture of fresh tomatoes and say, "Today's limited-time sale, from 3:00 to 5:00 PM."

[1370] Server-based information analysis

[1371] The server receives the image and audio data sent from the device. It uses image recognition technology such as OpenCV or TensorFlow to extract product information (product name, price) from the received image data. For example, it analyzes the text in the image and obtains the necessary information from the product label and price display. It also converts the audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the time of the limited-time sale and other special information.

[1372] Generate flyers

[1373] The server aggregates product information and supplementary information collected from all stores. It compiles the data from each store in a unified format and organizes it as information for the entire shopping district. For example, it automatically generates electronic flyers based on the data from each store, appropriately positioning product names, prices, and sale information. The generated electronic flyers are output in PDF or image format.

[1374] Electronic distribution

[1375] The server obtains the LINE account information of pre-registered users. Once the digital flyer is generated, it is distributed using the LINE API. Users can easily check the latest daily information on their smartphones or tablets.

[1376] User Emotion Recognition

[1377] The server uses an emotion engine to analyze the user's reaction when they receive and view the electronic flyer. The emotion engine can recognize emotions by analyzing the user's facial expressions and voice comments when they view the flyer. For example, if a user smiles when they view the flyer or leaves an interesting voice comment, that information is collected as emotion data.

[1378] Information Optimization

[1379] The server stores the emotion data recognized by the emotion engine and reflects it in the next flyer distribution. This allows the server to identify products and information that users are interested in and customize the content of the next flyer. For example, if a particular product received a good response in the previous distribution, the server can prioritize the distribution of information similar to that product.

[1380] Examples and prompts

[1381] For example, when a store inputs information that "fresh tomatoes are on sale at a special price," the store clerk takes a photo of the tomato and records the price information. Additionally, a voice message provides additional information, such as "Today's limited-time sale, from 3:00 PM to 5:00 PM." The device sends this information to a server, which extracts the information through image and voice recognition. The server then generates a digital flyer based on the aggregated data and distributes it to the user via LINE. When the user receives the flyer, the emotion engine analyzes the user's reaction and optimizes the content of future deliveries.

[1382] An example of a prompt is as follows:

[1383] "We collect image and audio data of the 'Today's Recommended Products' provided by each store in the shopping district, and use image and audio recognition technology to extract product information and limited-time sale information. We then compile all of the information into a unified format to generate digital flyers and distribute them to users using the LINE API. We analyze users' emotional reactions and incorporate feedback into the next distribution."

[1384] The above is a specific embodiment for carrying out the present invention. This system enables each store in a shopping mall to efficiently distribute discount information and analyze user reactions to optimize the content of future distributions.

[1385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1386] Step 1:

[1387] Information gathering and input data generation

[1388] Using the device's camera, the store clerk takes a picture of the "Today's Recommended Product" and saves it as an image file. Similarly, using the device's microphone, the store clerk records audio of the product description and supplementary information about the limited-time sale, and saves it as an audio file. In actual operation, for example, a store clerk can take a picture of a fresh tomato and explain in audio, "Today's limited-time sale, from 3:00 PM to 5:00 PM." The input for this step is the product image and audio of the product description, and the output is an image file and an audio file.

[1389] Step 2:

[1390] Sending and receiving data to the server

[1391] The device sends the generated image and audio files to the server. The server receives these data and prepares them for analysis. The input in this step is the data sent from the device (image and audio files), and the output is these files stored on the server.

[1392] Step 3:

[1393] Processing image data and extracting product information

[1394] The server uses image recognition technology to extract product information (product name, price) from the received image file. For example, it uses tools such as OpenCV or TensorFlow to analyze the text in the image and obtain the necessary information from the product label and price display. The input for this process is the image file, and the output is the extracted product name and price information.

[1395] Step 4:

[1396] Processing audio data and extracting supplementary information

[1397] The server converts the received audio file into text using speech recognition technology such as the Google Cloud Speech-to-Text API, and extracts the sale time and other special information. The input in this step is the audio file, and the output is the text supplementary information.

[1398] Step 5:

[1399] Generate electronic flyers

[1400] The server aggregates the product information and supplementary information collected from all stores and automatically generates an electronic flyer. Specifically, it organizes the collected data into a unified format and arranges the product name, price, and supplementary information in an appropriate layout. The input in this step is the product information and supplementary information collected from each store, and the output is an electronic flyer in PDF or image format.

[1401] Step 6:

[1402] Electronic distribution

[1403] The server sends the generated e-flyer to the user's pre-registered LINE account. It distributes it to the user's smartphone or tablet using the LINE API. The input in this step is the generated e-flyer, and the output is the e-flyer distributed to the user's device.

[1404] Step 7:

[1405] User sentiment analysis

[1406] The server uses an emotion engine to analyze the user's reactions while they are viewing the electronic flyer. Specifically, it analyzes the user's facial expressions and voice comments to obtain emotional information. The input in this step is the user's viewing data (facial expressions, voice comments), and the output is the analyzed emotional data.

[1407] Step 8:

[1408] Information Optimization

[1409] The server saves the emotion data obtained by the emotion engine and reflects it in future flyer content. It analyzes past data to identify products and information that the user was particularly interested in, and customizes the content of the next flyer based on that. The input in this step is the analyzed emotion data, and the output is an optimized version of the next digital flyer.

[1410] The above is the specific flow of the program processing of this system.

[1411] (Application example 2)

[1412] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1413] Traditional shopping districts and retail stores lacked efficient methods for advertising discount information and seasonal products, resulting in insufficient communication of information to consumers and challenges in promoting store visits and sales. Furthermore, there was a lack of methods for collecting and analyzing consumer responses in real time and reflecting them in the next marketing strategy, making it difficult to respond quickly to consumer needs. This made it difficult to effectively attract customers amid intensifying competition between stores.

[1414] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1415] In this invention, the server includes a means for the terminal to capture and record image and audio data, a means for analyzing the data received by the server, a means for analyzing user reactions using an emotion engine, and a means for optimizing the content of the next flyer. This enables shopping malls and retail stores to efficiently distribute information to consumers and analyze their reactions, enabling them to respond quickly to consumer needs.

[1416] "Participating store terminal" refers to equipment used by each participating store to take product images and record audio.

[1417] "Today's Recommended Products" are products that are especially recommended on that day, and refer to products or services that the store wants to promote in a special way.

[1418] "Means for taking images" refers to the function of taking images of products using a device with a camera function and saving them as data.

[1419] "Means of recording audio" refers to the ability to record product descriptions and supplementary information about limited-time sales using a device with a microphone function and save it as audio data.

[1420] A "server" is a central processing unit that analyzes image and audio data and performs various processes.

[1421] The "means for receiving" refers to the function of receiving data transmitted from a terminal.

[1422] "Image recognition" refers to the technology of automatically analyzing and extracting characters and objects from image data.

[1423] "Product information" refers to data that indicates the product name, price, and other related information.

[1424] "Speech recognition" refers to the technology of converting voice data into text data and analyzing it.

[1425] "Additional Information" refers to sale times and other special information.

[1426] "Means for automatically generating flyers" refers to a function that automatically generates visually appealing flyers based on collected data.

[1427] "Means of electronic distribution" refers to the ability to distribute the generated flyer via email or messaging apps.

[1428] "User" refers to a consumer who wishes to receive electronic flyers and has registered in advance.

[1429] "Emotion engine" refers to technology that analyzes emotions from users' facial expressions and feedback comments.

[1430] "Means for analyzing reactions" refers to a function that uses an emotion engine to analyze the reactions of users when they view a flyer.

[1431] "Optimization means" refers to the function of adjusting the content of future flyers based on analyzed response data, and providing optimal information tailored to consumer preferences.

[1432] The present invention is a system for enabling shopping malls and retail stores to efficiently distribute discount information to consumers and analyze their reactions. The system includes a terminal, a server, and an emotion engine for analyzing user reactions.

[1433] 1. Information collection via devices

[1434] The terminal is a device operated by a store clerk at a participating store, and can be a smartphone or tablet. The clerk uses the device's camera to take pictures of the "Today's Recommended Products" and also records audio descriptions of the products and supplementary information about the limited-time sale. This data is temporarily stored on the terminal and later sent to the server.

[1435] 2. Information analysis by the server

[1436] The server receives the image and audio data sent from the device and performs the main analysis process. The image data is converted into text using image recognition software (e.g., Amazon Rekognition) to extract product name and price information. Similarly, the audio data is converted into text using speech recognition software (e.g., Google Cloud Speech-to-Text) to extract limited-time sale times and other special information.

[1437] 3. Generate flyers

[1438] The server aggregates product information and supplemental information collected from all stores and automatically generates visually appealing digital flyers using template generation tools such as Adobe InDesign and Canva API. The generated flyers are saved in PDF format.

[1439] 4. Electronic Delivery

[1440] The server then distributes the generated digital flyers to the LINE accounts of pre-registered users, using the LINE Messaging API, allowing users to check the latest information on their smartphones, tablets, or other devices.

[1441] 5. User Emotion Recognition

[1442] The server uses an emotion engine (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer) to analyze the user's reaction when viewing the flyer. It analyzes the user's facial expressions and feedback comments to recognize positive, negative, and neutral emotions.

[1443] 6. Information optimization

[1444] The server then optimizes the content of the next flyer based on the analyzed emotion data. This allows the display of product information that the user is interested in to be prioritized, improving advertising effectiveness. For example, if the user shows interest in fresh tomatoes, the next flyer will prominently feature information about fresh vegetables and special offers.

[1445] Specific examples

[1446] For example, if a store is selling fresh tomatoes at a special price, a store clerk can take a picture of the tomato with their smartphone and record a voice description of the product, saying, "Today only, special price on sale from 3:00 PM to 5:00 PM." This data is sent to a server and analyzed. The generated flyer is then sent to users via LINE, and the emotion engine analyzes their reactions. As a result, the content of the next flyer can be optimized more effectively.

[1447] Prompt Sentence Examples

[1448] Image analysis prompt:

[1449] Image file name: store_product.jpg

[1450] API used: Image recognition software

[1451] Information to be recognized: Product name, price

[1452] Prompt for speech analysis:

[1453] Audio file name: product_description.wav

[1454] API used: Speech recognition software

[1455] Extracted information: Time of sale, special information

[1456] Emotion recognition prompt:

[1457] User feedback: "These tomatoes look fresh and delicious."

[1458] API used: Emotion recognition software

[1459] Emotions to extract: positive, negative, neutral

[1460] By combining these technologies, shopping districts and retail stores can provide consumers with discount information at the optimal time, enabling effective marketing.

[1461] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1462] Step 1:

[1463] The device takes a picture of the "Today's Recommended Product." The input is a product image taken using the device's camera function, and the output is saved as an image file in the device's storage. A store clerk takes a picture of the product using the device's camera and saves the image.

[1464] Step 2:

[1465] The device records audio for product descriptions and supplementary information about limited-time sales. The input is audio recorded through a microphone, and the output is saved as an audio file in the device's storage. A store clerk explains the product into the microphone, and the audio is saved.

[1466] Step 3:

[1467] The device sends the captured images and recorded audio data to the server. The input is the image and audio files captured and recorded on the device, and the output is obtained by transferring this data to the server. The device uploads the data to the server via the network.

[1468] Step 4:

[1469] The server analyzes the image data it receives. The input is the image file received by the server, and the output is product information extracted from the text and objects in the image. The server analyzes the image using image recognition software (e.g., Amazon Rekognition) and extracts product name and price information.

[1470] Step 5:

[1471] The server analyzes the received audio data. The input is the audio file received by the server, and the output is supplementary information converted from the audio into text. The server uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the audio data into text and analyze it.

[1472] Step 6:

[1473] The server aggregates product information and supplementary information collected from all stores and automatically generates flyers. The input is the analyzed product information and supplementary information, and the output is an electronic flyer. The server uses a template generation tool (e.g., Adobe InDesign, Canva API) to generate flyers based on the acquired data.

[1474] Step 7:

[1475] The server distributes the generated e-flyer to pre-registered users. The input is the generated e-flyer and the user's contact information, and the output is the flyer distributed to the recipient user. The server uses the LINE Messaging API to send the flyer to the user's LINE account.

[1476] Step 8:

[1477] The server analyzes the user's reaction when viewing the flyer. The input is the user's facial expression data and feedback comments, and the output is the user's emotion recognition results. The server analyzes the user's reaction using emotion recognition software (e.g., Microsoft Azure Emotion API, IBM Watson Tone Analyzer).

[1478] Step 9:

[1479] The server optimizes the content of the next flyer based on the analysis results. The input is the analyzed emotion data, and the output is the optimized flyer content. Based on the feedback obtained, the server prioritizes product information that matches the user's preferences and reflects it in the next flyer.

[1480] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1481] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1482] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1483] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1484] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1485] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1486] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1487] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1488] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1489] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1490] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1491] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1492] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1493] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1494] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1495] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1496] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1497] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1498] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1499] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1500] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1501] The following is further disclosed regarding the above embodiment.

[1502] (Claim 1)

[1503] A means for the terminals at participating stores to take pictures of "Today's Recommended Products";

[1504] The device will record audio information about product descriptions and limited-time sales,

[1505] A means for the server to receive image and audio data transmitted from the terminal;

[1506] A means for the server to extract product information using image recognition;

[1507] means for the server to extract supplemental information from the voice data using voice recognition;

[1508] A means for aggregating the information of all stores acquired by the server and automatically generating flyers;

[1509] means for electronically distributing flyers generated by the server to pre-registered users;

[1510] A system including:

[1511] (Claim 2)

[1512] 10. The system of claim 1, wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

[1513] (Claim 3)

[1514] 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition.

[1515] "Example 1"

[1516] (Claim 1)

[1517] A means for the terminals at participating stores to take pictures of "Today's Recommended Products";

[1518] The device will record audio information about product descriptions and limited-time sales,

[1519] means for transmitting image data and audio data collected by the terminal to a server;

[1520] A means for the server to receive image and audio data transmitted from the terminal;

[1521] A means for the server to extract product information using image recognition;

[1522] means for the server to extract supplemental information from the voice data using voice recognition;

[1523] A means for aggregating the information of all stores acquired by the server and automatically generating flyers in electronic document format;

[1524] means for electronically distributing flyers generated by the server to pre-registered users;

[1525] A system including:

[1526] (Claim 2)

[1527] 10. The system of claim 1, wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

[1528] (Claim 3)

[1529] 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition.

[1530] "Application Example 1"

[1531] (Claim 1)

[1532] A means for the terminals at participating stores to take pictures of "Today's Recommended Products";

[1533] The device will record audio information about product descriptions and limited-time sales,

[1534] A means for the server to receive image and audio data transmitted from the terminal;

[1535] A means for the server to extract product information using image recognition;

[1536] means for the server to extract supplemental information from the voice data using voice recognition;

[1537] A means for aggregating the information of all stores acquired by the server and automatically generating flyers;

[1538] means for electronically distributing flyers generated by the server to pre-registered users;

[1539] A means for smart glasses to display in-store product information in real time;

[1540] a means for the smart glasses to provide additional information about the product based on a voice command;

[1541] A means for the smart glasses to navigate to the shelves;

[1542] A system including:

[1543] (Claim 2)

[1544] 10. The system of claim 1, wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

[1545] (Claim 3)

[1546] 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition.

[1547] "Example 2: Combining Emotion Engines"

[1548] (Claim 1)

[1549] A means for the terminals at participating stores to take pictures of "Today's Recommended Products";

[1550] The device will record audio information about product descriptions and limited-time sales,

[1551] A means for the server to receive image and audio data transmitted from the terminal;

[1552] A means for the server to extract product information using image recognition;

[1553] means for the server to extract supplemental information from the voice data using voice recognition;

[1554] A means for aggregating the information of all stores acquired by the server and automatically generating flyers;

[1555] means for electronically distributing flyers generated by the server to pre-registered users;

[1556] A means for the server to analyze the user's reaction while viewing the flyer;

[1557] The server will use the analysis results to optimize the content of the next flyer,

[1558] A system including:

[1559] (Claim 2)

[1560] 10. The system of claim 1, wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

[1561] (Claim 3)

[1562] 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition.

[1563] "Application example 2 when combining emotion engines"

[1564] (Claim 1)

[1565] A means for the terminals at participating stores to take pictures of "Today's Recommended Products";

[1566] The device will record audio information about product descriptions and limited-time sales,

[1567] A means for the server to receive image and audio data transmitted from the terminal;

[1568] A means for the server to extract product information using image recognition;

[1569] means for the server to extract supplemental information from the voice data using voice recognition;

[1570] A means for aggregating information from all stores acquired by the server and automatically generating flyers;

[1571] A means for electronically distributing flyers generated by the server to pre-registered users;

[1572] A means for the server to analyze the user's reaction when viewing the flyer,

[1573] The server will use the user's response data to optimize the content of the next flyer.

[1574] A system including:

[1575] (Claim 2)

[1576] 10. The system of claim 1, wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

[1577] (Claim 3)

[1578] 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition. [Explanation of symbols]

[1579] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for the terminals at participating stores to take pictures of "Today's Recommended Products"; The device will record audio information about product descriptions and limited-time sales, A means for the server to receive image and audio data transmitted from the terminal; A means for the server to extract product information using image recognition; means for the server to extract supplemental information from the voice data using voice recognition; A means for aggregating the information of all stores acquired by the server and automatically generating flyers; means for electronically distributing flyers generated by the server to pre-registered users; A system including:

2. The system of claim 1 , wherein the server further comprises means for identifying product name and price information based on the image data to extract product information using image recognition.

3. 10. The system of claim 1, wherein the server further comprises means for parsing sale times and other special information to extract supplemental information from the voice data using voice recognition.

Images