System

Abstract

A system is provided.SOLUTION: This system is provided with a means for photographing the merchandise and price information of a store, a means for recording voice data, a means for recognizing and extracting the merchandise name and price information from the photographed image data, a means for analyzing special price information and time sale information from the recorded voice data, a means for collecting the recognized, extracted and analyzed information in an interface format, and a means for distributing an interface to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve the problem that each store in a shopping mall has few means to effectively disseminate information about price competitiveness and special offers, causing consumers to overlook the appeal of the products in the shopping mall. Another object of the present invention is to provide a method for each store in a shopping mall to provide consumers with the latest bargain information at low cost and efficiently, in order to compete with large supermarkets. [Means for solving the problem]

[0005] The present invention provides a system that includes the following means: a means for photographing product and price information in a store, a means for recording voice data, a means for recognizing and extracting product names and price information from the photographed image data, a means for analyzing special offer information and limited-time sale information from the recorded voice data, a means for compiling the recognized, extracted, and analyzed information in an interface format, and a means for delivering the interface to users. This makes it possible to effectively provide product information and special offer information in a store to consumers, thereby improving the competitiveness of the entire shopping district.

[0006] The "means for photographing products and price information in a store" refers to a device or method for photographing images and price information of products in a store and acquiring them as digital data.

[0007] "Voice data recording means" means a device or method for recording voice input by a merchant or employee as digital data.

[0008] "Means for recognizing and extracting product name and price information from captured image data" refers to technology that uses a specific algorithm to automatically extract and identify product name and price information from captured image data.

[0009] The "means for analyzing special offer information and limited time sale information from recorded audio data" refers to a technology for analyzing audio data, converting the content into text format, and identifying special offer information and limited time sale information.

[0010] "Means for compiling recognized, extracted, and analyzed information in an interface format" refers to technology that integrates extracted product information and analyzed special offer information and compiles them in a format that is easy for users to understand (for example, in the form of a flyer).

[0011] "Means for delivering the interface to users" refers to the technology used to transmit and deliver the generated information to users via a communications network.

[0012] A "terminal application" is software used by a store owner or employee to input voice data, and has the function of uploading the recorded voice data to a server.

[0013] A "communications network" is an infrastructure for transmitting and receiving data, including the Internet and local area networks.

[0014] An "image recognition algorithm" is a computer program that automatically identifies and extracts specific information (e.g., text or objects) from digital image data.

[0015] A "speech recognition algorithm" is a computer program that converts voice data into text data and analyzes its content.

[0016] A "means for automatically generating a flyer-format document" is a software or hardware method for automatically creating a flyer-format document based on collected data.

[0017] A "messaging application" is software for sending generated information to users in the form of messages, and includes, for example, LINE and WhatsApp.

[0018] A "drone or ceiling camera" is a device used to take images of products from inside a store or from above, allowing for efficient capture of a wide area.

[0019] A "database" is a system for systematically storing and managing various collected data. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0042] Data collection

[0043] server

[0044] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[0045] Drone / Ceiling Camera

[0046] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[0047] Shop owner (user)

[0048] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[0049] Data analysis

[0050] server

[0051] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[0052] Flyer generation

[0053] server

[0054] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[0055] Information distribution

[0056] server

[0057] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE).

[0058] Specific examples

[0059] As an example, an embodiment in a shopping mall is shown below.

[0060] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[0061] 2. The captured image data is sent to the server in real time.

[0062] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0063] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0064] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0065] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0066] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently and at low cost, thereby increasing its appeal to local consumers.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] server

[0070] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m. A shooting schedule is generated and the drone flight route is set based on the location information of the specified store. The ceiling camera also sets the timing to operate at the scheduled time.

[0071] Step 2:

[0072] Drone / Ceiling Camera

[0073] The drone and ceiling camera follow instructions from the server to capture images of products and prices at designated times and locations, and the captured image data is sent to the server in real time.

[0074] Step 3:

[0075] server

[0076] The server stores the received image data and uses image recognition algorithms to extract product names and prices. Image analysis identifies product labels and price tags and stores them in a database in text format.

[0077] Step 4:

[0078] Shop owner (user)

[0079] Using a dedicated terminal application, the store owner enters "Today's Deals" by voice at 10 a.m. The terminal application records the voice data and uploads it to the server in real time.

[0080] Step 5:

[0081] server

[0082] The server stores the received voice data and converts it into text using a speech recognition algorithm. Through speech analysis, special offers and limited-time sales information are extracted and stored in a database in text format.

[0083] Step 6:

[0084] server

[0085] The server integrates product information, price information, and special offer information stored in the database and automatically generates flyers containing "Today's Deals" for the entire shopping district. It creates visually easy-to-understand PDF files according to templates.

[0086] Step 7:

[0087] server

[0088] The server retrieves registered user information to distribute the generated flyer to nearby residents, and then distributes the flyer to the user at 11:00 AM using a messaging application (e.g., LINE).

[0089] Step 8:

[0090] User

[0091] Users can check the received message and view the flyer-style "Today's Deals," which allows them to keep up with the latest deals in the shopping district and shop efficiently.

[0092] In this way, the system of the present invention automatically collects and analyzes product information provided by each store in the shopping district, making it possible to efficiently provide consumers with the latest special price information.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] In the past, in order to quickly and efficiently communicate product information and special offers offered by each store in a shopping district to consumers, this was often done manually, which was time-consuming and costly. Furthermore, paper-based flyers had a large environmental impact and information updates were often delayed. Therefore, there was a growing need for an automated information provision system that could be updated in real time.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for photographing product and price information in stores using a drone and a ceiling camera and collecting the photographed data, means for recording voice data using a terminal application for speech input by the store owner, means for recognizing and extracting product names and price information from the collected image data using an image recognition algorithm, means for analyzing special offer information and limited-time sale information from the recorded voice data using a speech recognition algorithm, means for automatically generating a flyer-style interface using the recognized, extracted, and analyzed information, and means for distributing the generated flyer-style interface to registered users via a communication network. This enables the latest information on each store in the shopping district to be communicated to consumers quickly and efficiently.

[0098] A "drone" is an unmanned aerial vehicle that is flown by remote control or autonomous control and used to capture images of products and pricing information.

[0099] A "ceiling camera" is a fixed or movable camera device that is installed in a store and used to take images of shelves and merchandise in the store.

[0100] A "terminal application" is a device containing a software program used by a merchant to input voice data.

[0101] A "server" is a computer system that centrally manages data collection, analysis, storage, and distribution.

[0102] An "image recognition algorithm" is a software process for automatically extracting product name and price information from received image data.

[0103] "Speech recognition algorithm" means a software process that analyzes voice data provided by a merchant and converts it into textual information.

[0104] A "flyer-style interface" is a digital document that integrates product information and special offers and presents them in a visually easy-to-understand manner.

[0105] "Communications network" is a data transmission system used to distribute the generated flyer-style interface to registered users.

[0106] MODE FOR CARRYING OUT THE INVENTION

[0107] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0108] Data collection

[0109] server

[0110] The server controls the means of periodically collecting product information from each store in the shopping mall, including scheduling the drones and ceiling cameras, and sends shooting instructions to these devices.

[0111] Drone / Ceiling Camera

[0112] The drones and ceiling cameras follow instructions from the server and take images of products and price information for each store. The captured image data is sent to the server in real time. Specifically, drones equipped with high-resolution cameras fly over the shopping district, while ceiling cameras periodically take images of product shelves inside the stores. These devices instantly upload the data to the server via the network.

[0113] Shop owner (user)

[0114] The store owner uses a dedicated terminal application to provide "Today's Deals" by voice input. The voice data is automatically uploaded to the server. For example, "Apples are half price from 1pm to 2pm today."

[0115] Data analysis

[0116] server

[0117] The server stores the received image data and uses an image recognition algorithm to extract product name and price information. Specifically, it performs image analysis using libraries such as TensorFlow and OpenCV. It also uses a voice recognition algorithm (for example, a cloud-based voice recognition service) to convert the received voice data into text and analyzes the content. This allows it to extract information about special offers and limited-time sales. All analyzed data is stored in a database.

[0118] Flyer generation

[0119] server

[0120] Based on the information stored in the database, the server automatically generates flyers containing "Today's Deals" for the entire shopping district. This includes a program that integrates product information, price information, and special offers, and outputs the information in a format that is easy for users to understand visually. Specifically, the server uses tools such as Adobe InDesign Server to visually organize the integrated data and generate PDF files of the flyers.

[0121] Information distribution

[0122] server

[0123] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The distributed message includes a link to the flyer and a preview image, allowing users to easily check the contents.

[0124] Specific examples

[0125] As an example, an embodiment in a shopping mall is shown below.

[0126] 1. At 9:00 a.m., the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and photographs the products and prices of each store. At the same time, the ceiling camera photographs the product shelves inside the store.

[0127] 2. The captured image data is sent to the server in real time.

[0128] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0129] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0130] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0131] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0132] Prompt Sentence Examples

[0133] "This system collects product information from each store in the shopping district, automatically creates flyers, and distributes them to local residents. Products are photographed using drones and ceiling cameras, and the images are analyzed using TensorFlow and OpenCV. Store owners provide special offer information via voice input, which is converted into text using a speech recognition algorithm. This information is integrated and flyers are generated using Adobe InDesign Server. Finally, the flyers are distributed via the LINE Messaging API, making it easy for residents to access great deals."

[0134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0135] Step 1: Prepare for data collection

[0136] server

[0137] The server begins preparing for data collection at 8:00 a.m. every day. It checks the schedule for each store and sets the operation schedule for the drones and ceiling cameras. The server then checks that each device is working properly. At this stage, the server confirms that all equipment is operating correctly and prepares for the next step.

[0138] Input: Shopping street store information, device operation status

[0139] Output: Shooting instructions and schedule settings for each device

[0140] Step 2: Photograph your product information

[0141] server

[0142] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m., specifically detailed instructions including shooting points and flight routes. The drone flies over the shopping district, photographing products and pricing information at each store. The ceiling camera periodically photographs the product shelves inside the store.

[0143] Drone / Ceiling Camera

[0144] The drone uses a high-resolution camera to capture images of products and prices in stores, while a ceiling camera captures images of the shelves inside the store. The captured image data is sent to a server in real time.

[0145] Input: Shooting instruction from the server

[0146] Output: Image data (product and price information)

[0147] Step 3: Recording audio data

[0148] Shop owner (user)

[0149] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might say, "Apples are half price today from 1:00 p.m. to 2:00 p.m." This information is then automatically uploaded to the server.

[0150] Input: Shopkeeper's voice data (discount information)

[0151] Output: Audio data file

[0152] Step 4: Analyzing the image data

[0153] server

[0154] The server stores the received image data and uses image recognition algorithms to extract product name and price information. Specifically, it analyzes the image using libraries such as TensorFlow and OpenCV and converts the product name and price information into digital data.

[0155] Input: Captured image data

[0156] Output: Extracted product and price information

[0157] Step 5: Analyzing the audio data

[0158] server

[0159] The server converts the received voice data into text using a voice recognition algorithm. Specifically, it uses a voice recognition service to analyze the voice data and converts the special offers and limited-time sale information into text data.

[0160] Input: Audio data file

[0161] Output: Extracted special offers and deals

[0162] Step 6: Integrate data and generate flyers

[0163] server

[0164] The server integrates the analyzed product information, price information, and special offer information, automatically generating a flyer-style interface, and using software such as Adobe InDesign Server to visually organize the integrated data and generate a PDF file of the flyer.

[0165] Input: Product information, price information, special offers

[0166] Output: Automatically generated flyer format PDF file

[0167] Step 7: Distribute flyers

[0168] server

[0169] The server obtains registered user information to distribute the generated flyers to nearby residents. The flyers are distributed via messages using the LINE Messaging API, etc. The distributed messages include a link to the flyer and a preview image, allowing users to easily check the contents.

[0170] Input: Automatically generated flyer format PDF file, user information

[0171] Output: Distributed flyers and messages

[0172] Through these steps, product information and special offer information collected from each store in the shopping district can be efficiently analyzed and integrated, and quickly delivered to consumers.

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] It is difficult to provide users with the latest store product information and special offers in real time. Furthermore, manually collecting information and generating flyers takes time and effort, making it inefficient. Furthermore, it is important that the information delivered is visually easy to understand and delivered in a timely manner. New methods are needed to solve these problems.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes means for photographing product and price information from a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited-time sale information from the recorded voice data, means for compiling the recognized, extracted, and analyzed information in an interface format, means for generating this interface in a format viewable by users, and means for transmitting the generated interface via email or messaging service. This makes it possible to efficiently collect product and price information from a store in real time and provide it to users in a timely manner in a visually easy-to-understand format.

[0178] A "store" is a physical location for offering goods and services.

[0179] "Products" are goods and services sold to consumers.

[0180] "Price information" is information about the selling price of a product.

[0181] "Photographing means" refers to a device or system for capturing images of products.

[0182] "Audio data" means audio recorded in digital format.

[0183] A "recording means" is a device or system that records audio data.

[0184] "Photographed image data" refers to digital data of an image captured by a photographing device such as a camera.

[0185] The "product name" is a name that enables identification and recognition of the product.

[0186] "Means of recognition and extraction" refers to technologies and devices for identifying and extracting specific information from data.

[0187] "Special offer information" is information about products offered at special prices.

[0188] "Time sale information" is information about products for which special offers such as discounts are applied during specific time periods.

[0189] "Analytical means" are techniques and devices used to analyze data and interpret information.

[0190] An "interface format" is a display format that makes it easy for users to use information.

[0191] "Generative means" refers to the technology and devices used to create new information and forms based on data.

[0192] "Email" is a means of communication for sending and receiving messages over the Internet.

[0193] "Messaging Service" means a communication service for the real-time exchange of text and multimedia messages.

[0194] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments will be described in detail below.

[0195] System Configuration

[0196] 1. Information gathering

[0197] How to take photos of store products and pricing information:

[0198] The server uses a camera or a smart eyeglasses or other imaging device to acquire product and price information from each store, thereby collecting image data of each store's products.

[0199] Methods for recording audio data:

[0200] Shop owners use a dedicated terminal application to input information about special offers and limited-time sales by voice. Voice input is done on a smartphone or a dedicated terminal, and the recorded voice data is automatically uploaded to a server.

[0201] 2. Data Analysis

[0202] Methods for recognizing and extracting product name and price information from image data:

[0203] The server uses image recognition algorithms such as Google Cloud Vision API to extract product name and price information from the captured image data.

[0204] How to analyze special offers and time-limited sales information from recorded audio data:

[0205] The server uses speech recognition algorithms such as the Google Cloud Speech-to-Text API to convert the recorded audio data into text, and then analyzes the content to extract special offers and limited-time sales information.

[0206] 3. Flyer generation

[0207] Means of organizing the recognized, extracted and analyzed information in an interface format:

[0208] The server integrates the information stored in the database and generates flyers in a format that is easy for users to understand visually. It uses libraries such as FPDF to automatically generate flyers in PDF format.

[0209] A means of producing the generated interface in a user-viewable format:

[0210] The flyers are printed in a format that can be viewed on smartphones or smart glasses.

[0211] 4. Information distribution

[0212] How to send the generated interface via email or messaging services:

[0213] The server then distributes the generated flyer to users via email or messaging services (e.g., LINE), allowing users to receive the latest special offers and time-limited sales information in real time.

[0214] Examples:

[0215] For example, an example implementation in a shopping mall is shown below. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera to capture images of products and prices at each store. At the same time, the ceiling camera captures images of the store's shelves. At 10:00 AM, the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the voice data could include "Apples are half price from 1:00 PM to 2:00 PM today." The server then integrates the recognized and extracted information and automatically generates a PDF file in flyer format at 11:00 AM. At 12:00 PM, the server distributes the generated flyer to registered users using a messaging application.

[0216] Example prompt sentence:

[0217] 1. Take a photo of the product.

[0218] 2. Enter product information and special offers using your voice.

[0219] 3. Perform image and voice recognition and integrate the extracted information into a flyer format.

[0220] 4. Review the generated flyer and make any necessary corrections.

[0221] 5. The completed flyer will be sent to registered users via email or LINE.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] The server sends a shooting instruction to the drone and ceiling camera at 9:00 AM. The input is the schedule data for the previous day, and the output is a command to start shooting. The server sends this command to the drone and camera, causing them to take pictures of the store's products and price information.

[0225] Step 2:

[0226] The terminal uploads the captured image data to the server in real time. The input is the image data captured by the drone and ceiling camera, and the output is the image data uploaded to the server. The terminal then completes the collection of image data.

[0227] Step 3:

[0228] The server sends a notification to the merchant for voice input at 10:00 AM. The input is the preset notification schedule and the output is sending the notification to the merchant. The server then prompts the merchant to input voice data.

[0229] Step 4:

[0230] The merchant uses a dedicated terminal application to input "Today's Deals" by voice. The input is the merchant's voice, and the output is voice data stored on the terminal. The merchant then uploads this voice data to the terminal.

[0231] Step 5:

[0232] The terminal uploads the recorded voice data to the server in real time. The input is the voice data recorded by the merchant, and the output is the voice data uploaded to the server.

[0233] Step 6:

[0234] The server uses the Google Cloud Vision API to recognize and extract product names and price information from image data. The input is the uploaded image data, and the output is text data with the product names and price information extracted. The server stores this data in a database.

[0235] Step 7:

[0236] The server uses the Google Cloud Speech-to-Text API to convert the recorded voice data into text and analyze the special offer and time sale information. The input is the uploaded voice data, and the output is the analyzed text data of the special offer and time sale information. The server stores this data in a database.

[0237] Step 8:

[0238] The server integrates product information, price information, special offer information, and time sale information stored in the database, and automatically generates a PDF file in flyer format using the FPDF library. The input is the information stored in the database, and the output is the generated PDF flyer.

[0239] Step 9:

[0240] The server distributes the generated PDF flyer to registered users via email or messaging services. The input is the generated PDF file and the user's contact information, and the output is the flyer sent to the user. This allows users to receive the latest information on shopping mall deals in real time.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] The present invention relates to a system that automatically collects product information provided by each store in a shopping mall, and creates and distributes the information in the form of a flyer while taking into consideration the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[0243] Data collection

[0244] server

[0245] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[0246] Drone / Ceiling Camera

[0247] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[0248] Shop owner (user)

[0249] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[0250] Data analysis

[0251] server

[0252] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[0253] Flyer generation

[0254] server

[0255] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[0256] Emotion Engine

[0257] server

[0258] The server is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's voice data and facial expression data and evaluates the user's emotions in real time. Based on this data, the emotion engine provides an interface and recommendation information that is adapted to the user's emotional state.

[0259] Information distribution

[0260] server

[0261] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The content of the interface is adjusted based on the user's emotional state by an emotion engine.

[0262] Specific examples

[0263] As an example, an embodiment in a shopping mall is shown below.

[0264] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[0265] 2. The captured image data is sent to the server in real time.

[0266] 3. The store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, they can record voice data such as "Apples are half price from 1:00 PM to 2:00 PM today."

[0267] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0268] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0269] 6. Before distributing the generated flyer to nearby residents, the server analyzes the user's emotional state using an emotion engine. For example, if the server detects that the user has recently been feeling stressed, it will provide an interface containing recommendations for soothing products and positive messages.

[0270] 7. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0271] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently at low cost, thereby not only increasing the store's appeal to local consumers but also enabling the store to provide information according to the user's emotional state.

[0272] The processing flow will be explained below.

[0273] Step 1:

[0274] server

[0275] The server sends a shooting command to the drone and ceiling camera at 9:00 a.m. This prepares each device to start shooting information for the designated store. The server sets the drone's flight route and shooting timing based on the shooting schedule and location information.

[0276] Step 2:

[0277] Drone / Ceiling Camera

[0278] The drones and ceiling cameras capture images of store products and pricing information according to a set schedule and location information. For example, the drone flies over a shopping district and captures images of each store's storefront, while the ceiling camera captures images of the product shelves inside the store. The captured image data is sent to a server in real time.

[0279] Step 3:

[0280] server

[0281] The server receives the captured image data and temporarily stores it in storage. It then uses image recognition algorithms to extract product names and price information from the image data. For example, it identifies product labels and price tags and converts them into text data. The extracted information is then stored in a database.

[0282] Step 4:

[0283] Shop owner (user)

[0284] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might input information like, "Apples are half price today from 1:00 p.m. to 2:00 p.m." The voice data is uploaded to the server in real time.

[0285] Step 5:

[0286] server

[0287] The server stores the received voice data and converts it into text using a speech recognition algorithm. The voice data is analyzed to extract special offers and limited-time sales information, which is then converted into text and stored in a database.

[0288] Step 6:

[0289] server

[0290] The server integrates product information, price information, and special offers stored in the database and automatically generates a flyer with "Today's Deals." The flyer is created in a visually easy-to-understand PDF format.

[0291] Step 7:

[0292] server

[0293] Before delivering the flyer, the server uses an emotion engine to evaluate the user's emotional state. It analyzes the user's voice and facial expression data to recognize their emotional state in real time. For example, if the user is feeling stressed through voice analysis, it will consider recommending relaxation products accordingly.

[0294] Step 8:

[0295] server

[0296] The generated flyer information is then adjusted based on the emotion engine's evaluation results. For example, designs and messages that stimulate positive emotions are added. The final flyer is then distributed to nearby residents at 11:00 AM.

[0297] Step 9:

[0298] server

[0299] The server retrieves registered user information and distributes flyers via messaging applications, such as LINE, to users, sending flyers in PDF format.

[0300] Step 10:

[0301] User

[0302] The user checks the received message and browses the flyer-style "Today's Deals." This allows the user to keep up with the latest deals from the shopping mall and check recommended products that are adapted to their emotional state.

[0303] In this way, the system of the present invention can automatically collect and analyze product information provided by each store in the shopping district, generate flyers, and provide information optimized based on the user's emotional state.

[0304] Example 2

[0305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] Conventional systems have had difficulty collecting product information provided by stores in shopping districts and delivering that information appropriately, taking into account the user's emotional state. In particular, there was a need for a method to effectively aggregate product information and automate the delivery of information that reflects the user's emotional state. Furthermore, there was a need for a system that would allow store owners themselves to easily add information.

[0307] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing product and price information of stores in a commercial district, a means for recording voice data, a means for recognizing and extracting product names and price information from the photographed image data, a means for analyzing special offer information and limited-time sale information from the recorded voice data, a means for compiling the recognized, extracted, and analyzed information in an interface format, a means for recognizing the user's emotional state, a means for adjusting the generated interface based on the user's emotional state, and a means for delivering the interface to the user. This makes it possible to efficiently collect product information offered by each store in the shopping district and deliver information in the form of a flyer that takes the user's emotional state into consideration.

[0308] A "commercial district" is a specific area where multiple stores are located and where shopping and services are provided.

[0309] "Store" means a facility or place of business established to offer goods or services.

[0310] "Product and price information" is information about the names of products sold in the store and their selling prices.

[0311] "Means for photographing" refers to a method or apparatus for acquiring image data using a device such as a camera or drone.

[0312] "Voice data" refers to information collected through voice input, and is data that includes a particular human voice.

[0313] A "recording means" is a method or device for recording or capturing audio data.

[0314] "Image data" refers to visual information captured by a camera or drone stored in digital format.

[0315] "Means for recognition and extraction" refers to methods or devices that automatically detect and extract specific information from images or sounds.

[0316] "Special offer information" is information about products or services that are discounted below their regular price.

[0317] "Time sale information" is information about discount sales that take place during specific time periods.

[0318] "Means of analysis" are methods or devices for analyzing collected information and finding specific meaning or value.

[0319] An "interface" is a display format or means for providing information in a format that is visually easy for users to understand.

[0320] "User emotional state" refers to the user's emotional state, such as stress level or happiness.

[0321] A "means for recognizing" is a method or device for detecting and evaluating a user's emotions or state.

[0322] A "means for adjusting" is a method or device for optimizing information or display content based on the user's particular condition.

[0323] "Distribution means" refers to a method or device that uses communication means or a network to deliver information to users.

[0324] The present invention is a system that automatically collects product information offered by stores in a commercial area, and creates and distributes it in the form of a flyer, taking into account the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[0325] Data collection

[0326] server

[0327] The server manages the schedules of the drones and ceiling cameras to collect product information from each store in the commercial district. The server sends shooting instructions to the drones and ceiling cameras every day at 9:00 a.m. The devices operate according to these instructions and capture images of the store's products and price information.

[0328] Drone / Ceiling Camera

[0329] The drone and ceiling camera operate under instructions from the server. The drone photographs the store's exterior and merchandise, while the ceiling camera photographs the store's shelves. The captured image data is sent to the server in real time.

[0330] User

[0331] The store owner (user) uses a dedicated terminal application to input "Today's Deals" by voice. As an example of voice input, the terminal application records information such as "Apples are half price from 1pm to 2pm today," and the voice data is sent to the server.

[0332] Data analysis

[0333] server

[0334] The server analyzes the received image data using an image recognition algorithm to extract product name and price information. For example, the name of an "apple" and the price information "100 yen" are automatically extracted from the image data. The server also converts the received voice data into text using a voice recognition algorithm, and analyzes special offer information and limited time sale information. Through this analysis, special offer information such as "Apples are half price" is obtained as text.

[0335] Flyer generation

[0336] server

[0337] Based on the information stored in the database, the server automatically generates flyers with "Today's Deals" for the entire commercial district. The flyers are output as PDF files that display product images, names, prices, and special offers in a visually easy-to-understand format.

[0338] Emotion Engine

[0339] server

[0340] The server is equipped with an emotion engine that analyzes the user's voice and facial expression data to assess their emotional state. For example, if it determines from past data that the user has recently been feeling stressed, it will provide an interface containing products with a relaxing effect and positive messages.

[0341] Information distribution

[0342] server

[0343] The generated flyer is distributed to nearby residents via a messaging application (e.g., LINE). The server obtains registered user information, adjusts the interface content of the flyer based on the user's emotional state using an emotion engine, and distributes it at 12:00 a.m. Users can check the flyer from the received message and find out about the latest deals.

[0344] Specific examples

[0345] 1. The server sends a schedule to the drone and ceiling camera at 9:00 AM to "start filming."

[0346] 2. The drone flies over the commercial area, taking pictures of products and prices in each store and sending the images to the server. At the same time, a ceiling camera takes pictures of products inside the store and sends them to the server.

[0347] 3. The shopkeeper uses a dedicated terminal application to voice-input "Apples are half price." The voice data is uploaded to the server.

[0348] 4. The server analyzes the image data using an image recognition algorithm to extract product names and prices. It also converts the voice data into text using a voice recognition program to extract special offers and limited-time sales information.

[0349] 5. The server consolidates the analyzed information and automatically generates a flyer-format PDF file at 11:00 AM.

[0350] 6. The server uses an emotion engine to analyze past voice and facial expression data and adjust the flyer content based on the user's emotional state.

[0351] 7. At 12:00 AM, the server uses a messaging application to distribute the generated flyer to nearby residents. Users can then view the flyer in the message they receive and get the latest deals.

[0352] Example prompts for generative AI models

[0353] "Please explain in detail the specific steps involved in the system that automatically collects product information from each store in a commercial area, creates and distributes flyers in the form of flyers while taking into account the user's emotional state, including data collection, analysis of product information, flyer generation, and the function of the emotion engine."

[0354] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0355] Step 1: Prepare for data collection

[0356] server

[0357] The server sets a shooting schedule for the drone and ceiling camera at 9:00 AM every day and sends a shooting instruction. The server sets the schedule based on the current date and time as input for this instruction. As output, it generates an instruction for the drone and ceiling camera to start shooting. Based on this shooting instruction, the devices prepare to move on to the next step.

[0358] Step 2: Photograph your product information

[0359] Drone / Ceiling Camera

[0360] The drone and ceiling camera receive shooting instructions from the server and take pictures of products and price information in the commercial area. When receiving shooting instructions from the server as input, the drone takes pictures of the store's exterior and products in store, and the ceiling camera takes pictures of the product shelves inside the store. As output, the captured image data is generated and sent to the server in real time.

[0361] Specifically, the drone will fly over commercial areas along designated flight routes, taking high-resolution images of the exteriors of each store and the products on display, while the ceiling camera will continuously capture images of the shelves from a fixed position inside the store.

[0362] Step 3: Enter merchant information

[0363] User

[0364] The shop owner uses a dedicated terminal application to input "Today's Deals" by voice. As input, the shop owner enters voice data such as "Apples are half price today from 1 PM to 2 PM" into the terminal application. As output, this voice data is sent from the terminal to the server.

[0365] Specifically, the merchant launches the app, presses the voice input button on the screen, and then speaks the information. The voice is recorded by the app and instantly uploaded to the server.

[0366] Step 4: Collect image and audio data

[0367] server

[0368] The server receives and stores the captured image data and the audio data sent by the merchant. As input, the image data and audio data are sent to the server. As output, these data are stored in a temporary storage area within the server.

[0369] Specifically, as each piece of data is received by the server's data storage system, it is saved as a file in the appropriate directory, along with metadata such as the date and time of the photo and store information.

[0370] Step 5: Parse product information

[0371] server

[0372] The server uses an image recognition algorithm to extract product names and price information from the collected image data. It also uses a voice recognition algorithm to convert the merchant's voice data into text and analyze it. The saved image data and voice data are given as input. The analyzed text data, including product names, price information, and special offers, is generated as output.

[0373] Specifically, the server launches image recognition software to detect, for example, the product name "apple" and the price information "100 yen" from the image. Similarly, voice recognition software converts voice data such as "Apples are half price from 1:00 PM to 2:00 PM" into text and extracts it as special offer information.

[0374] Step 6: Store in the database

[0375] server

[0376] The server stores the parsed product information, price information, special offer information, and time sale information in a database. The parsed text data is used as input. The information stored in the database is generated as output.

[0377] Specifically, the server's database management system inserts the analysis results into the appropriate tables and associates each product data.

[0378] Step 7: Auto-generate flyers

[0379] server

[0380] The server automatically generates flyers containing "Today's Deals" for the entire commercial district based on the information stored in the database. As input, it accesses relevant information from the database. As output, it generates a visually easy-to-understand PDF file in flyer format.

[0381] Specifically, the server uses a proprietary layout algorithm to combine product images, names, and price information to generate a formatted PDF file.

[0382] Step 8: Analyzing the user's emotional state

[0383] server

[0384] The server uses an emotion engine to analyze the user's voice data and facial expression data and evaluate their emotional state. The user's past voice data and facial expression data are used as input. The output is evaluation data about the user's current emotional state.

[0385] Specifically, the server runs a machine learning model to calculate the user's stress level and happiness based on the input data.

[0386] Step 9: Prepare your flyer for distribution

[0387] server

[0388] The server adjusts the content of the flyer generated based on the results of the emotion engine and creates a message to be delivered. The flyer PDF and emotion rating data are used as input. The adjusted interface content message is generated as output.

[0389] Specifically, the server creates a message with content most suitable for the user in a specified format via the adaptive messaging system.

[0390] Step 10: Distribute flyers

[0391] server

[0392] The server distributes the generated flyer to nearby residents using a messaging application (e.g., LINE) at 12:00 AM. As input, the message with the adjusted interface content is used. As output, the flyer distributed through the messaging app is generated.

[0393] Specifically, the server uses a messaging API to send a message to all registered users, who can then click on the message to view the flyer.

[0394] (Application example 2)

[0395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0396] In today's commercial environment, it is difficult to efficiently aggregate product information provided by stores and effectively distribute it to local consumers. Furthermore, there is a need to stimulate greater purchasing motivation by providing appropriate information tailored to consumers' emotional state. However, existing systems cannot meet these requirements and lack a means to quickly and effectively convey the latest special offers from shopping districts and individual stores to consumers. Therefore, a new system is needed to improve the consumer purchasing experience.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0398] In this invention, the server includes means for photographing product and price information in a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited time sale information from the recorded voice data, means for analyzing the user's emotional state, means for compiling the recognized, extracted, and analyzed information in an interface format adapted to the user's emotional state, and means for delivering the interface to the user. This makes it possible to quickly and effectively deliver the latest special offer information to consumers and to provide information tailored to the user's emotional state.

[0399] A "store" is a physical location for selling goods and services.

[0400] "Price information" is information about the selling price of a product.

[0401] "Photographing means" refers to a means of acquiring image data using a photographing device such as a camera or drone.

[0402] "Audio data" is digital data that contains a recording of a human voice.

[0403] "Product name" is the name given to a product that is distributed in the market.

[0404] "Recognition and extraction means" refers to the technical means for recognizing and extracting specific information from images and sounds.

[0405] "Special Offer Information" is information about special prices that are lower than the regular price.

[0406] "Time sale information" is information about sales that are only held during specific time periods.

[0407] An "emotional state" is the psychological state that a user is feeling at a given time.

[0408] "Analysis means" refers to means for analyzing and extracting information from digital data.

[0409] An "interface" is a display screen or operating means for exchanging information between a user and a computer system.

[0410] A "communications network" is an infrastructure for sending and receiving data between computers and devices.

[0411] A "delivery method" is a technique or method for delivering specific information to users.

[0412] The present invention relates to a system for efficiently collecting and analyzing product and price information from a store and delivering the information in an interface format that corresponds to the emotional state of a user. An embodiment of the system will be described in detail below.

[0413] Data collection

[0414] The server controls drones and ceiling cameras as a means of capturing images of store products and price information. These image capturing devices follow instructions from the server to collect image data including product and price information and send it to the server in real time. Store owners use a dedicated terminal application to record voice data. For example, they can input special offer information or limited-time sale information by voice, such as "Apples are half price from 1:00 PM to 2:00 PM today."

[0415] Data analysis

[0416] The server uses an image recognition algorithm (e.g., Python's OpenCV library) to analyze the received image data, thereby recognizing and extracting product names and prices, and storing them in a database. The voice data is converted into text using a speech recognition algorithm (e.g., Google Speech-to-Text API), and the server analyzes the special offers and limited-time sales information.

[0417] User sentiment analysis

[0418] Using the emotion engine, the server analyzes the user's voice data and facial expression data to evaluate their emotional state. The emotion engine uses a generative AI model (e.g., a Transformers model). This allows the server to recognize the user's emotional state, such as "feeling stressed" or "feeling happy," in real time.

[0419] Flyer generation and distribution

[0420] The server integrates the analyzed product information, price information, and special offer information to generate a flyer in an interface format that corresponds to the user's emotional state. The generated flyer is then distributed to users via a communication network. Specifically, it is distributed through a messaging application (e.g., LINE). In this way, consumers can obtain the latest special offer information in real time, increasing their motivation to purchase.

[0421] Examples and prompts

[0422] For example, at 9 a.m. in a shopping district, instructions to take product photos are sent from the server to drones and ceiling cameras. The store owner uses a dedicated terminal app to record voice data such as, "Apples are half price today from 1 p.m. to 2 p.m.." The data collected in this way is analyzed by the server and automatically generated as a flyer-style PDF file at 11 a.m.

[0423] The message the user receives will include the prompt:

[0424] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[0425] If the user has recently been identified as stressed:

[0426] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[0427] This will result in more personalized information and an improved consumer purchasing experience.

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1:

[0430] Data collection: The server sends shooting instructions to the drones and ceiling cameras. Input includes the location information and shooting schedule of each store. Based on this, the drones and ceiling cameras take pictures of products and prices, and send the image data to the server in real time. The store owner uses a terminal application to input special offers and limited-time sales information by voice. This voice data is also sent to the server.

[0431] Step 2:

[0432] Image data analysis: The server receives the captured image data and uses an image recognition algorithm (Python's OpenCV library) to recognize and extract product name and price information. The input is the captured image data, and the output is text data containing product name and price information. Through this process, the information obtained from the image is converted into text format and stored in a database.

[0433] Step 3:

[0434] Voice data analysis: The server receives voice data from the merchant and converts it into text using a voice recognition algorithm (Google Speech-to-Text API). The input is voice data, and the output is text data containing special offers and time sale information. Through this process, the special offer information obtained from the voice is converted into text format and stored in a database.

[0435] Step 4:

[0436] Emotion analysis: The server receives the user's voice data and facial expression data and uses an emotion engine to analyze the user's emotional state. A generative AI model (Transformers model) evaluates the user's emotions from the input voice data and facial expression data. The input is the user's voice data and facial expression data, and the output is text data that indicates the user's emotional state. This processing allows the user's psychological state to be accurately understood.

[0437] Step 5:

[0438] Flyer generation: The server integrates the analyzed product information, price information, and special offer information to generate a flyer that corresponds to the user's emotional state. The input is product information, price information, special offer information, and the user's emotional state, and the output is an emotion-adaptive flyer. This flyer is formatted in a visually easy-to-understand format and contains information appropriate for the user. This process visualizes the purchasing information that is most suitable for the user.

[0439] Step 6:

[0440] Information distribution: The server distributes the generated flyer to the user over a communication network. Specifically, the flyer is sent via a messaging application. The input is the generated flyer data, and the output is the flyer received by the user. This process allows users to quickly receive the latest special offers and emotion-adaptive purchasing information.

[0441] Example prompt sentence:

[0442] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[0443] If the user has recently been identified as stressed:

[0444] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0446] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0448] [Second embodiment]

[0449] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0450] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0451] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0453] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0455] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0456] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0457] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0460] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0461] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0462] Data collection

[0463] server

[0464] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[0465] Drone / Ceiling Camera

[0466] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[0467] Shop owner (user)

[0468] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[0469] Data analysis

[0470] server

[0471] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[0472] Flyer generation

[0473] server

[0474] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[0475] Information distribution

[0476] server

[0477] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE).

[0478] Specific examples

[0479] As an example, an embodiment in a shopping mall is shown below.

[0480] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[0481] 2. The captured image data is sent to the server in real time.

[0482] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0483] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0484] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0485] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0486] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently and at low cost, thereby increasing its appeal to local consumers.

[0487] The processing flow will be explained below.

[0488] Step 1:

[0489] server

[0490] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m. A shooting schedule is generated and the drone flight route is set based on the location information of the specified store. The ceiling camera also sets the timing to operate at the scheduled time.

[0491] Step 2:

[0492] Drone / Ceiling Camera

[0493] The drone and ceiling camera follow instructions from the server to capture images of products and prices at designated times and locations, and the captured image data is sent to the server in real time.

[0494] Step 3:

[0495] server

[0496] The server stores the received image data and uses image recognition algorithms to extract product names and prices. Image analysis identifies product labels and price tags and stores them in a database in text format.

[0497] Step 4:

[0498] Shop owner (user)

[0499] Using a dedicated terminal application, the store owner enters "Today's Deals" by voice at 10 a.m. The terminal application records the voice data and uploads it to the server in real time.

[0500] Step 5:

[0501] server

[0502] The server stores the received voice data and converts it into text using a speech recognition algorithm. Through speech analysis, special offers and limited-time sales information are extracted and stored in a database in text format.

[0503] Step 6:

[0504] server

[0505] The server integrates product information, price information, and special offer information stored in the database and automatically generates flyers containing "Today's Deals" for the entire shopping district. It creates visually easy-to-understand PDF files according to templates.

[0506] Step 7:

[0507] server

[0508] The server retrieves registered user information to distribute the generated flyer to nearby residents, and then distributes the flyer to the user at 11:00 AM using a messaging application (e.g., LINE).

[0509] Step 8:

[0510] User

[0511] Users can check the received message and view the flyer-style "Today's Deals," which allows them to keep up with the latest deals in the shopping district and shop efficiently.

[0512] In this way, the system of the present invention automatically collects and analyzes product information provided by each store in the shopping district, making it possible to efficiently provide consumers with the latest special price information.

[0513] Example 1

[0514] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0515] In the past, in order to quickly and efficiently communicate product information and special offers offered by each store in a shopping district to consumers, this was often done manually, which was time-consuming and costly. Furthermore, paper-based flyers had a large environmental impact and information updates were often delayed. Therefore, there was a growing need for an automated information provision system that could be updated in real time.

[0516] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0517] In this invention, the server includes means for photographing product and price information in stores using a drone and a ceiling camera and collecting the photographed data, means for recording voice data using a terminal application for speech input by the store owner, means for recognizing and extracting product names and price information from the collected image data using an image recognition algorithm, means for analyzing special offer information and limited-time sale information from the recorded voice data using a speech recognition algorithm, means for automatically generating a flyer-style interface using the recognized, extracted, and analyzed information, and means for distributing the generated flyer-style interface to registered users via a communication network. This enables the latest information on each store in the shopping district to be communicated to consumers quickly and efficiently.

[0518] A "drone" is an unmanned aerial vehicle that is flown by remote control or autonomous control and used to capture images of products and pricing information.

[0519] A "ceiling camera" is a fixed or movable camera device that is installed in a store and used to take images of shelves and merchandise in the store.

[0520] A "terminal application" is a device containing a software program used by a merchant to input voice data.

[0521] A "server" is a computer system that centrally manages data collection, analysis, storage, and distribution.

[0522] An "image recognition algorithm" is a software process for automatically extracting product name and price information from received image data.

[0523] "Speech recognition algorithm" means a software process that analyzes voice data provided by a merchant and converts it into textual information.

[0524] A "flyer-style interface" is a digital document that integrates product information and special offers and presents them in a visually easy-to-understand manner.

[0525] "Communications network" is a data transmission system used to distribute the generated flyer-style interface to registered users.

[0526] MODE FOR CARRYING OUT THE INVENTION

[0527] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0528] Data collection

[0529] server

[0530] The server controls the means of periodically collecting product information from each store in the shopping mall, including scheduling the drones and ceiling cameras, and sends shooting instructions to these devices.

[0531] Drone / Ceiling Camera

[0532] The drones and ceiling cameras follow instructions from the server and take images of products and price information for each store. The captured image data is sent to the server in real time. Specifically, drones equipped with high-resolution cameras fly over the shopping district, while ceiling cameras periodically take images of product shelves inside the stores. These devices instantly upload the data to the server via the network.

[0533] Shop owner (user)

[0534] The store owner uses a dedicated terminal application to provide "Today's Deals" by voice input. The voice data is automatically uploaded to the server. For example, "Apples are half price from 1pm to 2pm today."

[0535] Data analysis

[0536] server

[0537] The server stores the received image data and uses an image recognition algorithm to extract product name and price information. Specifically, it performs image analysis using libraries such as TensorFlow and OpenCV. It also uses a voice recognition algorithm (for example, a cloud-based voice recognition service) to convert the received voice data into text and analyzes the content. This allows it to extract information about special offers and limited-time sales. All analyzed data is stored in a database.

[0538] Flyer generation

[0539] server

[0540] Based on the information stored in the database, the server automatically generates flyers containing "Today's Deals" for the entire shopping district. This includes a program that integrates product information, price information, and special offers, and outputs the information in a format that is easy for users to understand visually. Specifically, the server uses tools such as Adobe InDesign Server to visually organize the integrated data and generate PDF files of the flyers.

[0541] Information distribution

[0542] server

[0543] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The distributed message includes a link to the flyer and a preview image, allowing users to easily check the contents.

[0544] Specific examples

[0545] As an example, an embodiment in a shopping mall is shown below.

[0546] 1. At 9:00 a.m., the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and photographs the products and prices of each store. At the same time, the ceiling camera photographs the product shelves inside the store.

[0547] 2. The captured image data is sent to the server in real time.

[0548] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0549] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0550] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0551] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0552] Prompt Sentence Examples

[0553] "This system collects product information from each store in the shopping district, automatically creates flyers, and distributes them to local residents. Products are photographed using drones and ceiling cameras, and the images are analyzed using TensorFlow and OpenCV. Store owners provide special offer information via voice input, which is converted into text using a speech recognition algorithm. This information is integrated and flyers are generated using Adobe InDesign Server. Finally, the flyers are distributed via the LINE Messaging API, making it easy for residents to access great deals."

[0554] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0555] Step 1: Prepare for data collection

[0556] server

[0557] The server begins preparing for data collection at 8:00 a.m. every day. It checks the schedule for each store and sets the operation schedule for the drones and ceiling cameras. The server then checks that each device is working properly. At this stage, the server confirms that all equipment is operating correctly and prepares for the next step.

[0558] Input: Shopping street store information, device operation status

[0559] Output: Shooting instructions and schedule settings for each device

[0560] Step 2: Photograph your product information

[0561] server

[0562] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m., specifically detailed instructions including shooting points and flight routes. The drone flies over the shopping district, photographing products and pricing information at each store. The ceiling camera periodically photographs the product shelves inside the store.

[0563] Drone / Ceiling Camera

[0564] The drone uses a high-resolution camera to capture images of products and prices in stores, while a ceiling camera captures images of the shelves inside the store. The captured image data is sent to a server in real time.

[0565] Input: Shooting instruction from the server

[0566] Output: Image data (product and price information)

[0567] Step 3: Recording audio data

[0568] Shop owner (user)

[0569] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might say, "Apples are half price today from 1:00 p.m. to 2:00 p.m." This information is then automatically uploaded to the server.

[0570] Input: Shopkeeper's voice data (discount information)

[0571] Output: Audio data file

[0572] Step 4: Analyzing the image data

[0573] server

[0574] The server stores the received image data and uses image recognition algorithms to extract product name and price information. Specifically, it analyzes the image using libraries such as TensorFlow and OpenCV and converts the product name and price information into digital data.

[0575] Input: Captured image data

[0576] Output: Extracted product and price information

[0577] Step 5: Analyzing the audio data

[0578] server

[0579] The server converts the received voice data into text using a voice recognition algorithm. Specifically, it uses a voice recognition service to analyze the voice data and converts the special offers and limited-time sale information into text data.

[0580] Input: Audio data file

[0581] Output: Extracted special offers and deals

[0582] Step 6: Integrate data and generate flyers

[0583] server

[0584] The server integrates the analyzed product information, price information, and special offer information, automatically generating a flyer-style interface, and using software such as Adobe InDesign Server to visually organize the integrated data and generate a PDF file of the flyer.

[0585] Input: Product information, price information, special offers

[0586] Output: Automatically generated flyer format PDF file

[0587] Step 7: Distribute flyers

[0588] server

[0589] The server obtains registered user information to distribute the generated flyers to nearby residents. The flyers are distributed via messages using the LINE Messaging API, etc. The distributed messages include a link to the flyer and a preview image, allowing users to easily check the contents.

[0590] Input: Automatically generated flyer format PDF file, user information

[0591] Output: Distributed flyers and messages

[0592] Through these steps, product information and special offer information collected from each store in the shopping district can be efficiently analyzed and integrated, and quickly delivered to consumers.

[0593] (Application example 1)

[0594] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] It is difficult to provide users with the latest store product information and special offers in real time. Furthermore, manually collecting information and generating flyers takes time and effort, making it inefficient. Furthermore, it is important that the information delivered is visually easy to understand and delivered in a timely manner. New methods are needed to solve these problems.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0597] In this invention, the server includes means for photographing product and price information from a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited-time sale information from the recorded voice data, means for compiling the recognized, extracted, and analyzed information in an interface format, means for generating this interface in a format viewable by users, and means for transmitting the generated interface via email or messaging service. This makes it possible to efficiently collect product and price information from a store in real time and provide it to users in a timely manner in a visually easy-to-understand format.

[0598] A "store" is a physical location for offering goods and services.

[0599] "Products" are goods and services sold to consumers.

[0600] "Price information" is information about the selling price of a product.

[0601] "Photographing means" refers to a device or system for capturing images of products.

[0602] "Audio data" means audio recorded in digital format.

[0603] A "recording means" is a device or system that records audio data.

[0604] "Photographed image data" refers to digital data of an image captured by a photographing device such as a camera.

[0605] The "product name" is a name that enables identification and recognition of the product.

[0606] "Means of recognition and extraction" refers to technologies and devices for identifying and extracting specific information from data.

[0607] "Special offer information" is information about products offered at special prices.

[0608] "Time sale information" is information about products for which special offers such as discounts are applied during specific time periods.

[0609] "Analytical means" are techniques and devices used to analyze data and interpret information.

[0610] An "interface format" is a display format that makes it easy for users to use information.

[0611] "Generative means" refers to the technology and devices used to create new information and forms based on data.

[0612] "Email" is a means of communication for sending and receiving messages over the Internet.

[0613] "Messaging Service" means a communication service for the real-time exchange of text and multimedia messages.

[0614] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments will be described in detail below.

[0615] System Configuration

[0616] 1. Information gathering

[0617] How to take photos of store products and pricing information:

[0618] The server uses a camera or a smart eyeglasses or other imaging device to acquire product and price information from each store, thereby collecting image data of each store's products.

[0619] Methods for recording audio data:

[0620] Shop owners use a dedicated terminal application to input information about special offers and limited-time sales by voice. Voice input is done on a smartphone or a dedicated terminal, and the recorded voice data is automatically uploaded to a server.

[0621] 2. Data Analysis

[0622] Methods for recognizing and extracting product name and price information from image data:

[0623] The server uses image recognition algorithms such as Google Cloud Vision API to extract product name and price information from the captured image data.

[0624] How to analyze special offers and time-limited sales information from recorded audio data:

[0625] The server uses speech recognition algorithms such as the Google Cloud Speech-to-Text API to convert the recorded audio data into text, and then analyzes the content to extract special offers and limited-time sales information.

[0626] 3. Flyer generation

[0627] Means of organizing the recognized, extracted and analyzed information in an interface format:

[0628] The server integrates the information stored in the database and generates flyers in a format that is easy for users to understand visually. It uses libraries such as FPDF to automatically generate flyers in PDF format.

[0629] A means of producing the generated interface in a user-viewable format:

[0630] The flyers are printed in a format that can be viewed on smartphones or smart glasses.

[0631] 4. Information distribution

[0632] How to send the generated interface via email or messaging services:

[0633] The server then distributes the generated flyer to users via email or messaging services (e.g., LINE), allowing users to receive the latest special offers and time-limited sales information in real time.

[0634] Examples:

[0635] For example, an example implementation in a shopping mall is shown below. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera to capture images of products and prices at each store. At the same time, the ceiling camera captures images of the store's shelves. At 10:00 AM, the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the voice data could include "Apples are half price from 1:00 PM to 2:00 PM today." The server then integrates the recognized and extracted information and automatically generates a PDF file in flyer format at 11:00 AM. At 12:00 PM, the server distributes the generated flyer to registered users using a messaging application.

[0636] Example prompt sentence:

[0637] 1. Take a photo of the product.

[0638] 2. Enter product information and special offers using your voice.

[0639] 3. Perform image and voice recognition and integrate the extracted information into a flyer format.

[0640] 4. Review the generated flyer and make any necessary corrections.

[0641] 5. The completed flyer will be sent to registered users via email or LINE.

[0642] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0643] Step 1:

[0644] The server sends a shooting instruction to the drone and ceiling camera at 9:00 AM. The input is the schedule data for the previous day, and the output is a command to start shooting. The server sends this command to the drone and camera, causing them to take pictures of the store's products and price information.

[0645] Step 2:

[0646] The terminal uploads the captured image data to the server in real time. The input is the image data captured by the drone and ceiling camera, and the output is the image data uploaded to the server. The terminal then completes the collection of image data.

[0647] Step 3:

[0648] The server sends a notification to the merchant for voice input at 10:00 AM. The input is the preset notification schedule and the output is sending the notification to the merchant. The server then prompts the merchant to input voice data.

[0649] Step 4:

[0650] The merchant uses a dedicated terminal application to input "Today's Deals" by voice. The input is the merchant's voice, and the output is voice data stored on the terminal. The merchant then uploads this voice data to the terminal.

[0651] Step 5:

[0652] The terminal uploads the recorded voice data to the server in real time. The input is the voice data recorded by the merchant, and the output is the voice data uploaded to the server.

[0653] Step 6:

[0654] The server uses the Google Cloud Vision API to recognize and extract product names and price information from image data. The input is the uploaded image data, and the output is text data with the product names and price information extracted. The server stores this data in a database.

[0655] Step 7:

[0656] The server uses the Google Cloud Speech-to-Text API to convert the recorded voice data into text and analyze the special offer and time sale information. The input is the uploaded voice data, and the output is the analyzed text data of the special offer and time sale information. The server stores this data in a database.

[0657] Step 8:

[0658] The server integrates product information, price information, special offer information, and time sale information stored in the database, and automatically generates a PDF file in flyer format using the FPDF library. The input is the information stored in the database, and the output is the generated PDF flyer.

[0659] Step 9:

[0660] The server distributes the generated PDF flyer to registered users via email or messaging services. The input is the generated PDF file and the user's contact information, and the output is the flyer sent to the user. This allows users to receive the latest information on shopping mall deals in real time.

[0661] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0662] The present invention relates to a system that automatically collects product information provided by each store in a shopping mall, and creates and distributes the information in the form of a flyer while taking into consideration the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[0663] Data collection

[0664] server

[0665] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[0666] Drone / Ceiling Camera

[0667] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[0668] Shop owner (user)

[0669] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[0670] Data analysis

[0671] server

[0672] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[0673] Flyer generation

[0674] server

[0675] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[0676] Emotion Engine

[0677] server

[0678] The server is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's voice data and facial expression data and evaluates the user's emotions in real time. Based on this data, the emotion engine provides an interface and recommendation information that is adapted to the user's emotional state.

[0679] Information distribution

[0680] server

[0681] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The content of the interface is adjusted based on the user's emotional state by an emotion engine.

[0682] Specific examples

[0683] As an example, an embodiment in a shopping mall is shown below.

[0684] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[0685] 2. The captured image data is sent to the server in real time.

[0686] 3. The store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, they can record voice data such as "Apples are half price from 1:00 PM to 2:00 PM today."

[0687] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0688] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0689] 6. Before distributing the generated flyer to nearby residents, the server analyzes the user's emotional state using an emotion engine. For example, if the server detects that the user has recently been feeling stressed, it will provide an interface containing recommendations for soothing products and positive messages.

[0690] 7. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0691] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently at low cost, thereby not only increasing the store's appeal to local consumers but also enabling the store to provide information according to the user's emotional state.

[0692] The processing flow will be explained below.

[0693] Step 1:

[0694] server

[0695] The server sends a shooting command to the drone and ceiling camera at 9:00 a.m. This prepares each device to start shooting information for the designated store. The server sets the drone's flight route and shooting timing based on the shooting schedule and location information.

[0696] Step 2:

[0697] Drone / Ceiling Camera

[0698] The drones and ceiling cameras capture images of store products and pricing information according to a set schedule and location information. For example, the drone flies over a shopping district and captures images of each store's storefront, while the ceiling camera captures images of the product shelves inside the store. The captured image data is sent to a server in real time.

[0699] Step 3:

[0700] server

[0701] The server receives the captured image data and temporarily stores it in storage. It then uses image recognition algorithms to extract product names and price information from the image data. For example, it identifies product labels and price tags and converts them into text data. The extracted information is then stored in a database.

[0702] Step 4:

[0703] Shop owner (user)

[0704] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might input information like, "Apples are half price today from 1:00 p.m. to 2:00 p.m." The voice data is uploaded to the server in real time.

[0705] Step 5:

[0706] server

[0707] The server stores the received voice data and converts it into text using a speech recognition algorithm. The voice data is analyzed to extract special offers and limited-time sales information, which is then converted into text and stored in a database.

[0708] Step 6:

[0709] server

[0710] The server integrates product information, price information, and special offers stored in the database and automatically generates a flyer with "Today's Deals." The flyer is created in a visually easy-to-understand PDF format.

[0711] Step 7:

[0712] server

[0713] Before delivering the flyer, the server uses an emotion engine to evaluate the user's emotional state. It analyzes the user's voice and facial expression data to recognize their emotional state in real time. For example, if the user is feeling stressed through voice analysis, it will consider recommending relaxation products accordingly.

[0714] Step 8:

[0715] server

[0716] The generated flyer information is then adjusted based on the emotion engine's evaluation results. For example, designs and messages that stimulate positive emotions are added. The final flyer is then distributed to nearby residents at 11:00 AM.

[0717] Step 9:

[0718] server

[0719] The server retrieves registered user information and distributes flyers via messaging applications, such as LINE, to users, sending flyers in PDF format.

[0720] Step 10:

[0721] User

[0722] The user checks the received message and browses the flyer-style "Today's Deals." This allows the user to keep up with the latest deals from the shopping mall and check recommended products that are adapted to their emotional state.

[0723] In this way, the system of the present invention can automatically collect and analyze product information provided by each store in the shopping district, generate flyers, and provide information optimized based on the user's emotional state.

[0724] Example 2

[0725] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0726] Conventional systems have had difficulty collecting product information provided by stores in shopping districts and delivering that information appropriately, taking into account the user's emotional state. In particular, there was a need for a method to effectively aggregate product information and automate the delivery of information that reflects the user's emotional state. Furthermore, there was a need for a system that would allow store owners themselves to easily add information.

[0727] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing product and price information of stores in a commercial district, a means for recording voice data, a means for recognizing and extracting product names and price information from the photographed image data, a means for analyzing special offer information and limited-time sale information from the recorded voice data, a means for compiling the recognized, extracted, and analyzed information in an interface format, a means for recognizing the user's emotional state, a means for adjusting the generated interface based on the user's emotional state, and a means for delivering the interface to the user. This makes it possible to efficiently collect product information offered by each store in the shopping district and deliver information in the form of a flyer that takes the user's emotional state into consideration.

[0728] A "commercial district" is a specific area where multiple stores are located and where shopping and services are provided.

[0729] "Store" means a facility or place of business established to offer goods or services.

[0730] "Product and price information" is information about the names of products sold in the store and their selling prices.

[0731] "Means for photographing" refers to a method or apparatus for acquiring image data using a device such as a camera or drone.

[0732] "Voice data" refers to information collected through voice input, and is data that includes a particular human voice.

[0733] A "recording means" is a method or device for recording or capturing audio data.

[0734] "Image data" refers to visual information captured by a camera or drone stored in digital format.

[0735] "Means for recognition and extraction" refers to methods or devices that automatically detect and extract specific information from images or sounds.

[0736] "Special offer information" is information about products or services that are discounted below their regular price.

[0737] "Time sale information" is information about discount sales that take place during specific time periods.

[0738] "Means of analysis" are methods or devices for analyzing collected information and finding specific meaning or value.

[0739] An "interface" is a display format or means for providing information in a format that is visually easy for users to understand.

[0740] "User emotional state" refers to the user's emotional state, such as stress level or happiness.

[0741] A "means for recognizing" is a method or device for detecting and evaluating a user's emotions or state.

[0742] A "means for adjusting" is a method or device for optimizing information or display content based on the user's particular condition.

[0743] "Distribution means" refers to a method or device that uses communication means or a network to deliver information to users.

[0744] The present invention is a system that automatically collects product information offered by stores in a commercial area, and creates and distributes it in the form of a flyer, taking into account the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[0745] Data collection

[0746] server

[0747] The server manages the schedules of the drones and ceiling cameras to collect product information from each store in the commercial district. The server sends shooting instructions to the drones and ceiling cameras every day at 9:00 a.m. The devices operate according to these instructions and capture images of the store's products and price information.

[0748] Drone / Ceiling Camera

[0749] The drone and ceiling camera operate under instructions from the server. The drone photographs the store's exterior and merchandise, while the ceiling camera photographs the store's shelves. The captured image data is sent to the server in real time.

[0750] User

[0751] The store owner (user) uses a dedicated terminal application to input "Today's Deals" by voice. As an example of voice input, the terminal application records information such as "Apples are half price from 1pm to 2pm today," and the voice data is sent to the server.

[0752] Data analysis

[0753] server

[0754] The server analyzes the received image data using an image recognition algorithm to extract product name and price information. For example, the name of an "apple" and the price information "100 yen" are automatically extracted from the image data. The server also converts the received voice data into text using a voice recognition algorithm, and analyzes special offer information and limited time sale information. Through this analysis, special offer information such as "Apples are half price" is obtained as text.

[0755] Flyer generation

[0756] server

[0757] Based on the information stored in the database, the server automatically generates flyers with "Today's Deals" for the entire commercial district. The flyers are output as PDF files that display product images, names, prices, and special offers in a visually easy-to-understand format.

[0758] Emotion Engine

[0759] server

[0760] The server is equipped with an emotion engine that analyzes the user's voice and facial expression data to assess their emotional state. For example, if it determines from past data that the user has recently been feeling stressed, it will provide an interface containing products with a relaxing effect and positive messages.

[0761] Information distribution

[0762] server

[0763] The generated flyer is distributed to nearby residents via a messaging application (e.g., LINE). The server obtains registered user information, adjusts the interface content of the flyer based on the user's emotional state using an emotion engine, and distributes it at 12:00 a.m. Users can check the flyer from the received message and find out about the latest deals.

[0764] Specific examples

[0765] 1. The server sends a schedule to the drone and ceiling camera at 9:00 AM to "start filming."

[0766] 2. The drone flies over the commercial area, taking pictures of products and prices in each store and sending the images to the server. At the same time, a ceiling camera takes pictures of products inside the store and sends them to the server.

[0767] 3. The shopkeeper uses a dedicated terminal application to voice-input "Apples are half price." The voice data is uploaded to the server.

[0768] 4. The server analyzes the image data using an image recognition algorithm to extract product names and prices. It also converts the voice data into text using a voice recognition program to extract special offers and limited-time sales information.

[0769] 5. The server consolidates the analyzed information and automatically generates a flyer-format PDF file at 11:00 AM.

[0770] 6. The server uses an emotion engine to analyze past voice and facial expression data and adjust the flyer content based on the user's emotional state.

[0771] 7. At 12:00 AM, the server uses a messaging application to distribute the generated flyer to nearby residents. Users can then view the flyer in the message they receive and get the latest deals.

[0772] Example prompts for generative AI models

[0773] "Please explain in detail the specific steps involved in the system that automatically collects product information from each store in a commercial area, creates and distributes flyers in the form of flyers while taking into account the user's emotional state, including data collection, analysis of product information, flyer generation, and the function of the emotion engine."

[0774] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0775] Step 1: Prepare for data collection

[0776] server

[0777] The server sets a shooting schedule for the drone and ceiling camera at 9:00 AM every day and sends a shooting instruction. The server sets the schedule based on the current date and time as input for this instruction. As output, it generates an instruction for the drone and ceiling camera to start shooting. Based on this shooting instruction, the devices prepare to move on to the next step.

[0778] Step 2: Photograph your product information

[0779] Drone / Ceiling Camera

[0780] The drone and ceiling camera receive shooting instructions from the server and take pictures of products and price information in the commercial area. When receiving shooting instructions from the server as input, the drone takes pictures of the store's exterior and products in store, and the ceiling camera takes pictures of the product shelves inside the store. As output, the captured image data is generated and sent to the server in real time.

[0781] Specifically, the drone will fly over commercial areas along designated flight routes, taking high-resolution images of the exteriors of each store and the products on display, while the ceiling camera will continuously capture images of the shelves from a fixed position inside the store.

[0782] Step 3: Enter merchant information

[0783] User

[0784] The shop owner uses a dedicated terminal application to input "Today's Deals" by voice. As input, the shop owner enters voice data such as "Apples are half price today from 1 PM to 2 PM" into the terminal application. As output, this voice data is sent from the terminal to the server.

[0785] Specifically, the merchant launches the app, presses the voice input button on the screen, and then speaks the information. The voice is recorded by the app and instantly uploaded to the server.

[0786] Step 4: Collect image and audio data

[0787] server

[0788] The server receives and stores the captured image data and the audio data sent by the merchant. As input, the image data and audio data are sent to the server. As output, these data are stored in a temporary storage area within the server.

[0789] Specifically, as each piece of data is received by the server's data storage system, it is saved as a file in the appropriate directory, along with metadata such as the date and time of the photo and store information.

[0790] Step 5: Parse product information

[0791] server

[0792] The server uses an image recognition algorithm to extract product names and price information from the collected image data. It also uses a voice recognition algorithm to convert the merchant's voice data into text and analyze it. The saved image data and voice data are given as input. The analyzed text data, including product names, price information, and special offers, is generated as output.

[0793] Specifically, the server launches image recognition software to detect, for example, the product name "apple" and the price information "100 yen" from the image. Similarly, voice recognition software converts voice data such as "Apples are half price from 1:00 PM to 2:00 PM" into text and extracts it as special offer information.

[0794] Step 6: Store in the database

[0795] server

[0796] The server stores the parsed product information, price information, special offer information, and time sale information in a database. The parsed text data is used as input. The information stored in the database is generated as output.

[0797] Specifically, the server's database management system inserts the analysis results into the appropriate tables and associates each product data.

[0798] Step 7: Auto-generate flyers

[0799] server

[0800] The server automatically generates flyers containing "Today's Deals" for the entire commercial district based on the information stored in the database. As input, it accesses relevant information from the database. As output, it generates a visually easy-to-understand PDF file in flyer format.

[0801] Specifically, the server uses a proprietary layout algorithm to combine product images, names, and price information to generate a formatted PDF file.

[0802] Step 8: Analyzing the user's emotional state

[0803] server

[0804] The server uses an emotion engine to analyze the user's voice data and facial expression data and evaluate their emotional state. The user's past voice data and facial expression data are used as input. The output is evaluation data about the user's current emotional state.

[0805] Specifically, the server runs a machine learning model to calculate the user's stress level and happiness based on the input data.

[0806] Step 9: Prepare your flyer for distribution

[0807] server

[0808] The server adjusts the content of the flyer generated based on the results of the emotion engine and creates a message to be delivered. The flyer PDF and emotion rating data are used as input. The adjusted interface content message is generated as output.

[0809] Specifically, the server creates a message with content most suitable for the user in a specified format via the adaptive messaging system.

[0810] Step 10: Distribute flyers

[0811] server

[0812] The server distributes the generated flyer to nearby residents using a messaging application (e.g., LINE) at 12:00 AM. As input, the message with the adjusted interface content is used. As output, the flyer distributed through the messaging app is generated.

[0813] Specifically, the server uses a messaging API to send a message to all registered users, who can then click on the message to view the flyer.

[0814] (Application example 2)

[0815] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0816] In today's commercial environment, it is difficult to efficiently aggregate product information provided by stores and effectively distribute it to local consumers. Furthermore, there is a need to stimulate greater purchasing motivation by providing appropriate information tailored to consumers' emotional state. However, existing systems cannot meet these requirements and lack a means to quickly and effectively convey the latest special offers from shopping districts and individual stores to consumers. Therefore, a new system is needed to improve the consumer purchasing experience.

[0817] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0818] In this invention, the server includes means for photographing product and price information in a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited time sale information from the recorded voice data, means for analyzing the user's emotional state, means for compiling the recognized, extracted, and analyzed information in an interface format adapted to the user's emotional state, and means for delivering the interface to the user. This makes it possible to quickly and effectively deliver the latest special offer information to consumers and to provide information tailored to the user's emotional state.

[0819] A "store" is a physical location for selling goods and services.

[0820] "Price information" is information about the selling price of a product.

[0821] "Photographing means" refers to a means of acquiring image data using a photographing device such as a camera or drone.

[0822] "Audio data" is digital data that contains a recording of a human voice.

[0823] "Product name" is the name given to a product that is distributed in the market.

[0824] "Recognition and extraction means" refers to the technical means for recognizing and extracting specific information from images and sounds.

[0825] "Special Offer Information" is information about special prices that are lower than the regular price.

[0826] "Time sale information" is information about sales that are only held during specific time periods.

[0827] An "emotional state" is the psychological state that a user is feeling at a given time.

[0828] "Analysis means" refers to means for analyzing and extracting information from digital data.

[0829] An "interface" is a display screen or operating means for exchanging information between a user and a computer system.

[0830] A "communications network" is an infrastructure for sending and receiving data between computers and devices.

[0831] A "delivery method" is a technique or method for delivering specific information to users.

[0832] The present invention relates to a system for efficiently collecting and analyzing product and price information from a store and delivering the information in an interface format that corresponds to the emotional state of a user. An embodiment of the system will be described in detail below.

[0833] Data collection

[0834] The server controls drones and ceiling cameras as a means of capturing images of store products and price information. These image capturing devices follow instructions from the server to collect image data including product and price information and send it to the server in real time. Store owners use a dedicated terminal application to record voice data. For example, they can input special offer information or limited-time sale information by voice, such as "Apples are half price from 1:00 PM to 2:00 PM today."

[0835] Data analysis

[0836] The server uses an image recognition algorithm (e.g., Python's OpenCV library) to analyze the received image data, thereby recognizing and extracting product names and prices, and storing them in a database. The voice data is converted into text using a speech recognition algorithm (e.g., Google Speech-to-Text API), and the server analyzes the special offers and limited-time sales information.

[0837] User sentiment analysis

[0838] Using the emotion engine, the server analyzes the user's voice data and facial expression data to evaluate their emotional state. The emotion engine uses a generative AI model (e.g., a Transformers model). This allows the server to recognize the user's emotional state, such as "feeling stressed" or "feeling happy," in real time.

[0839] Flyer generation and distribution

[0840] The server integrates the analyzed product information, price information, and special offer information to generate a flyer in an interface format that corresponds to the user's emotional state. The generated flyer is then distributed to users via a communication network. Specifically, it is distributed through a messaging application (e.g., LINE). In this way, consumers can obtain the latest special offer information in real time, increasing their motivation to purchase.

[0841] Examples and prompts

[0842] For example, at 9 a.m. in a shopping district, instructions to take product photos are sent from the server to drones and ceiling cameras. The store owner uses a dedicated terminal app to record voice data such as, "Apples are half price today from 1 p.m. to 2 p.m.." The data collected in this way is analyzed by the server and automatically generated as a flyer-style PDF file at 11 a.m.

[0843] The message the user receives will include the prompt:

[0844] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[0845] If the user has recently been identified as stressed:

[0846] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[0847] This will result in more personalized information and an improved consumer purchasing experience.

[0848] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0849] Step 1:

[0850] Data collection: The server sends shooting instructions to the drones and ceiling cameras. Input includes the location information and shooting schedule of each store. Based on this, the drones and ceiling cameras take pictures of products and prices, and send the image data to the server in real time. The store owner uses a terminal application to input special offers and limited-time sales information by voice. This voice data is also sent to the server.

[0851] Step 2:

[0852] Image data analysis: The server receives the captured image data and uses an image recognition algorithm (Python's OpenCV library) to recognize and extract product name and price information. The input is the captured image data, and the output is text data containing product name and price information. Through this process, the information obtained from the image is converted into text format and stored in a database.

[0853] Step 3:

[0854] Voice data analysis: The server receives voice data from the merchant and converts it into text using a voice recognition algorithm (Google Speech-to-Text API). The input is voice data, and the output is text data containing special offers and time sale information. Through this process, the special offer information obtained from the voice is converted into text format and stored in a database.

[0855] Step 4:

[0856] Emotion analysis: The server receives the user's voice data and facial expression data and uses an emotion engine to analyze the user's emotional state. A generative AI model (Transformers model) evaluates the user's emotions from the input voice data and facial expression data. The input is the user's voice data and facial expression data, and the output is text data that indicates the user's emotional state. This processing allows the user's psychological state to be accurately understood.

[0857] Step 5:

[0858] Flyer generation: The server integrates the analyzed product information, price information, and special offer information to generate a flyer that corresponds to the user's emotional state. The input is product information, price information, special offer information, and the user's emotional state, and the output is an emotion-adaptive flyer. This flyer is formatted in a visually easy-to-understand format and contains information appropriate for the user. This process visualizes the purchasing information that is most suitable for the user.

[0859] Step 6:

[0860] Information distribution: The server distributes the generated flyer to the user over a communication network. Specifically, the flyer is sent via a messaging application. The input is the generated flyer data, and the output is the flyer received by the user. This process allows users to quickly receive the latest special offers and emotion-adaptive purchasing information.

[0861] Example prompt sentence:

[0862] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[0863] If the user has recently been identified as stressed:

[0864] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[0865] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0866] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0867] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0868] [Third embodiment]

[0869] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0870] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0871] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0872] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0873] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0874] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0875] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0876] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0877] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0878] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0879] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0880] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0881] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0882] Data collection

[0883] server

[0884] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[0885] Drone / Ceiling Camera

[0886] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[0887] Shop owner (user)

[0888] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[0889] Data analysis

[0890] server

[0891] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[0892] Flyer generation

[0893] server

[0894] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[0895] Information distribution

[0896] server

[0897] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE).

[0898] Specific examples

[0899] As an example, an embodiment in a shopping mall is shown below.

[0900] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[0901] 2. The captured image data is sent to the server in real time.

[0902] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0903] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0904] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0905] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0906] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently and at low cost, thereby increasing its appeal to local consumers.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] server

[0910] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m. A shooting schedule is generated and the drone flight route is set based on the location information of the specified store. The ceiling camera also sets the timing to operate at the scheduled time.

[0911] Step 2:

[0912] Drone / Ceiling Camera

[0913] The drone and ceiling camera follow instructions from the server to capture images of products and prices at designated times and locations, and the captured image data is sent to the server in real time.

[0914] Step 3:

[0915] server

[0916] The server stores the received image data and uses image recognition algorithms to extract product names and prices. Image analysis identifies product labels and price tags and stores them in a database in text format.

[0917] Step 4:

[0918] Shop owner (user)

[0919] Using a dedicated terminal application, the store owner enters "Today's Deals" by voice at 10 a.m. The terminal application records the voice data and uploads it to the server in real time.

[0920] Step 5:

[0921] server

[0922] The server stores the received voice data and converts it into text using a speech recognition algorithm. Through speech analysis, special offers and limited-time sales information are extracted and stored in a database in text format.

[0923] Step 6:

[0924] server

[0925] The server integrates product information, price information, and special offer information stored in the database and automatically generates flyers containing "Today's Deals" for the entire shopping district. It creates visually easy-to-understand PDF files according to templates.

[0926] Step 7:

[0927] server

[0928] The server retrieves registered user information to distribute the generated flyer to nearby residents, and then distributes the flyer to the user at 11:00 AM using a messaging application (e.g., LINE).

[0929] Step 8:

[0930] User

[0931] Users can check the received message and view the flyer-style "Today's Deals," which allows them to keep up with the latest deals in the shopping district and shop efficiently.

[0932] In this way, the system of the present invention automatically collects and analyzes product information provided by each store in the shopping district, making it possible to efficiently provide consumers with the latest special price information.

[0933] Example 1

[0934] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0935] In the past, in order to quickly and efficiently communicate product information and special offers offered by each store in a shopping district to consumers, this was often done manually, which was time-consuming and costly. Furthermore, paper-based flyers had a large environmental impact and information updates were often delayed. Therefore, there was a growing need for an automated information provision system that could be updated in real time.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0937] In this invention, the server includes means for photographing product and price information in stores using a drone and a ceiling camera and collecting the photographed data, means for recording voice data using a terminal application for speech input by the store owner, means for recognizing and extracting product names and price information from the collected image data using an image recognition algorithm, means for analyzing special offer information and limited-time sale information from the recorded voice data using a speech recognition algorithm, means for automatically generating a flyer-style interface using the recognized, extracted, and analyzed information, and means for distributing the generated flyer-style interface to registered users via a communication network. This enables the latest information on each store in the shopping district to be communicated to consumers quickly and efficiently.

[0938] A "drone" is an unmanned aerial vehicle that is flown by remote control or autonomous control and used to capture images of products and pricing information.

[0939] A "ceiling camera" is a fixed or movable camera device that is installed in a store and used to take images of shelves and merchandise in the store.

[0940] A "terminal application" is a device containing a software program used by a merchant to input voice data.

[0941] A "server" is a computer system that centrally manages data collection, analysis, storage, and distribution.

[0942] An "image recognition algorithm" is a software process for automatically extracting product name and price information from received image data.

[0943] "Speech recognition algorithm" means a software process that analyzes voice data provided by a merchant and converts it into textual information.

[0944] A "flyer-style interface" is a digital document that integrates product information and special offers and presents them in a visually easy-to-understand manner.

[0945] "Communications network" is a data transmission system used to distribute the generated flyer-style interface to registered users.

[0946] MODE FOR CARRYING OUT THE INVENTION

[0947] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[0948] Data collection

[0949] server

[0950] The server controls the means of periodically collecting product information from each store in the shopping mall, including scheduling the drones and ceiling cameras, and sends shooting instructions to these devices.

[0951] Drone / Ceiling Camera

[0952] The drones and ceiling cameras follow instructions from the server and take images of products and price information for each store. The captured image data is sent to the server in real time. Specifically, drones equipped with high-resolution cameras fly over the shopping district, while ceiling cameras periodically take images of product shelves inside the stores. These devices instantly upload the data to the server via the network.

[0953] Shop owner (user)

[0954] The store owner uses a dedicated terminal application to provide "Today's Deals" by voice input. The voice data is automatically uploaded to the server. For example, "Apples are half price from 1pm to 2pm today."

[0955] Data analysis

[0956] server

[0957] The server stores the received image data and uses an image recognition algorithm to extract product name and price information. Specifically, it performs image analysis using libraries such as TensorFlow and OpenCV. It also uses a voice recognition algorithm (for example, a cloud-based voice recognition service) to convert the received voice data into text and analyzes the content. This allows it to extract information about special offers and limited-time sales. All analyzed data is stored in a database.

[0958] Flyer generation

[0959] server

[0960] Based on the information stored in the database, the server automatically generates flyers containing "Today's Deals" for the entire shopping district. This includes a program that integrates product information, price information, and special offers, and outputs the information in a format that is easy for users to understand visually. Specifically, the server uses tools such as Adobe InDesign Server to visually organize the integrated data and generate PDF files of the flyers.

[0961] Information distribution

[0962] server

[0963] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The distributed message includes a link to the flyer and a preview image, allowing users to easily check the contents.

[0964] Specific examples

[0965] As an example, an embodiment in a shopping mall is shown below.

[0966] 1. At 9:00 a.m., the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and photographs the products and prices of each store. At the same time, the ceiling camera photographs the product shelves inside the store.

[0967] 2. The captured image data is sent to the server in real time.

[0968] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[0969] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[0970] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[0971] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[0972] Prompt Sentence Examples

[0973] "This system collects product information from each store in the shopping district, automatically creates flyers, and distributes them to local residents. Products are photographed using drones and ceiling cameras, and the images are analyzed using TensorFlow and OpenCV. Store owners provide special offer information via voice input, which is converted into text using a speech recognition algorithm. This information is integrated and flyers are generated using Adobe InDesign Server. Finally, the flyers are distributed via the LINE Messaging API, making it easy for residents to access great deals."

[0974] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0975] Step 1: Prepare for data collection

[0976] server

[0977] The server begins preparing for data collection at 8:00 a.m. every day. It checks the schedule for each store and sets the operation schedule for the drones and ceiling cameras. The server then checks that each device is working properly. At this stage, the server confirms that all equipment is operating correctly and prepares for the next step.

[0978] Input: Shopping street store information, device operation status

[0979] Output: Shooting instructions and schedule settings for each device

[0980] Step 2: Photograph your product information

[0981] server

[0982] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m., specifically detailed instructions including shooting points and flight routes. The drone flies over the shopping district, photographing products and pricing information at each store. The ceiling camera periodically photographs the product shelves inside the store.

[0983] Drone / Ceiling Camera

[0984] The drone uses a high-resolution camera to capture images of products and prices in stores, while a ceiling camera captures images of the shelves inside the store. The captured image data is sent to a server in real time.

[0985] Input: Shooting instruction from the server

[0986] Output: Image data (product and price information)

[0987] Step 3: Recording audio data

[0988] Shop owner (user)

[0989] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might say, "Apples are half price today from 1:00 p.m. to 2:00 p.m." This information is then automatically uploaded to the server.

[0990] Input: Shopkeeper's voice data (discount information)

[0991] Output: Audio data file

[0992] Step 4: Analyzing the image data

[0993] server

[0994] The server stores the received image data and uses image recognition algorithms to extract product name and price information. Specifically, it analyzes the image using libraries such as TensorFlow and OpenCV and converts the product name and price information into digital data.

[0995] Input: Captured image data

[0996] Output: Extracted product and price information

[0997] Step 5: Analyzing the audio data

[0998] server

[0999] The server converts the received voice data into text using a voice recognition algorithm. Specifically, it uses a voice recognition service to analyze the voice data and converts the special offers and limited-time sale information into text data.

[1000] Input: Audio data file

[1001] Output: Extracted special offers and deals

[1002] Step 6: Integrate data and generate flyers

[1003] server

[1004] The server integrates the analyzed product information, price information, and special offer information, automatically generating a flyer-style interface, and using software such as Adobe InDesign Server to visually organize the integrated data and generate a PDF file of the flyer.

[1005] Input: Product information, price information, special offers

[1006] Output: Automatically generated flyer format PDF file

[1007] Step 7: Distribute flyers

[1008] server

[1009] The server obtains registered user information to distribute the generated flyers to nearby residents. The flyers are distributed via messages using the LINE Messaging API, etc. The distributed messages include a link to the flyer and a preview image, allowing users to easily check the contents.

[1010] Input: Automatically generated flyer format PDF file, user information

[1011] Output: Distributed flyers and messages

[1012] Through these steps, product information and special offer information collected from each store in the shopping district can be efficiently analyzed and integrated, and quickly delivered to consumers.

[1013] (Application example 1)

[1014] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] It is difficult to provide users with the latest store product information and special offers in real time. Furthermore, manually collecting information and generating flyers takes time and effort, making it inefficient. Furthermore, it is important that the information delivered is visually easy to understand and delivered in a timely manner. New methods are needed to solve these problems.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1017] In this invention, the server includes means for photographing product and price information from a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited-time sale information from the recorded voice data, means for compiling the recognized, extracted, and analyzed information in an interface format, means for generating this interface in a format viewable by users, and means for transmitting the generated interface via email or messaging service. This makes it possible to efficiently collect product and price information from a store in real time and provide it to users in a timely manner in a visually easy-to-understand format.

[1018] A "store" is a physical location for offering goods and services.

[1019] "Products" are goods and services sold to consumers.

[1020] "Price information" is information about the selling price of a product.

[1021] "Photographing means" refers to a device or system for capturing images of products.

[1022] "Audio data" means audio recorded in digital format.

[1023] A "recording means" is a device or system that records audio data.

[1024] "Photographed image data" refers to digital data of an image captured by a photographing device such as a camera.

[1025] The "product name" is a name that enables identification and recognition of the product.

[1026] "Means of recognition and extraction" refers to technologies and devices for identifying and extracting specific information from data.

[1027] "Special offer information" is information about products offered at special prices.

[1028] "Time sale information" is information about products for which special offers such as discounts are applied during specific time periods.

[1029] "Analytical means" are techniques and devices used to analyze data and interpret information.

[1030] An "interface format" is a display format that makes it easy for users to use information.

[1031] "Generative means" refers to the technology and devices used to create new information and forms based on data.

[1032] "Email" is a means of communication for sending and receiving messages over the Internet.

[1033] "Messaging Service" means a communication service for the real-time exchange of text and multimedia messages.

[1034] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments will be described in detail below.

[1035] System Configuration

[1036] 1. Information gathering

[1037] How to take photos of store products and pricing information:

[1038] The server uses a camera or a smart eyeglasses or other imaging device to acquire product and price information from each store, thereby collecting image data of each store's products.

[1039] Methods for recording audio data:

[1040] Shop owners use a dedicated terminal application to input information about special offers and limited-time sales by voice. Voice input is done on a smartphone or a dedicated terminal, and the recorded voice data is automatically uploaded to a server.

[1041] 2. Data Analysis

[1042] Methods for recognizing and extracting product name and price information from image data:

[1043] The server uses image recognition algorithms such as Google Cloud Vision API to extract product name and price information from the captured image data.

[1044] How to analyze special offers and time-limited sales information from recorded audio data:

[1045] The server uses speech recognition algorithms such as the Google Cloud Speech-to-Text API to convert the recorded audio data into text, and then analyzes the content to extract special offers and limited-time sales information.

[1046] 3. Flyer generation

[1047] Means of organizing the recognized, extracted and analyzed information in an interface format:

[1048] The server integrates the information stored in the database and generates flyers in a format that is easy for users to understand visually. It uses libraries such as FPDF to automatically generate flyers in PDF format.

[1049] A means of producing the generated interface in a user-viewable format:

[1050] The flyers are printed in a format that can be viewed on smartphones or smart glasses.

[1051] 4. Information distribution

[1052] How to send the generated interface via email or messaging services:

[1053] The server then distributes the generated flyer to users via email or messaging services (e.g., LINE), allowing users to receive the latest special offers and time-limited sales information in real time.

[1054] Examples:

[1055] For example, an example implementation in a shopping mall is shown below. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera to capture images of products and prices at each store. At the same time, the ceiling camera captures images of the store's shelves. At 10:00 AM, the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the voice data could include "Apples are half price from 1:00 PM to 2:00 PM today." The server then integrates the recognized and extracted information and automatically generates a PDF file in flyer format at 11:00 AM. At 12:00 PM, the server distributes the generated flyer to registered users using a messaging application.

[1056] Example prompt sentence:

[1057] 1. Take a photo of the product.

[1058] 2. Enter product information and special offers using your voice.

[1059] 3. Perform image and voice recognition and integrate the extracted information into a flyer format.

[1060] 4. Review the generated flyer and make any necessary corrections.

[1061] 5. The completed flyer will be sent to registered users via email or LINE.

[1062] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1063] Step 1:

[1064] The server sends a shooting instruction to the drone and ceiling camera at 9:00 AM. The input is the schedule data for the previous day, and the output is a command to start shooting. The server sends this command to the drone and camera, causing them to take pictures of the store's products and price information.

[1065] Step 2:

[1066] The terminal uploads the captured image data to the server in real time. The input is the image data captured by the drone and ceiling camera, and the output is the image data uploaded to the server. The terminal then completes the collection of image data.

[1067] Step 3:

[1068] The server sends a notification to the merchant for voice input at 10:00 AM. The input is the preset notification schedule and the output is sending the notification to the merchant. The server then prompts the merchant to input voice data.

[1069] Step 4:

[1070] The merchant uses a dedicated terminal application to input "Today's Deals" by voice. The input is the merchant's voice, and the output is voice data stored on the terminal. The merchant then uploads this voice data to the terminal.

[1071] Step 5:

[1072] The terminal uploads the recorded voice data to the server in real time. The input is the voice data recorded by the merchant, and the output is the voice data uploaded to the server.

[1073] Step 6:

[1074] The server uses the Google Cloud Vision API to recognize and extract product names and price information from image data. The input is the uploaded image data, and the output is text data with the product names and price information extracted. The server stores this data in a database.

[1075] Step 7:

[1076] The server uses the Google Cloud Speech-to-Text API to convert the recorded voice data into text and analyze the special offer and time sale information. The input is the uploaded voice data, and the output is the analyzed text data of the special offer and time sale information. The server stores this data in a database.

[1077] Step 8:

[1078] The server integrates product information, price information, special offer information, and time sale information stored in the database, and automatically generates a PDF file in flyer format using the FPDF library. The input is the information stored in the database, and the output is the generated PDF flyer.

[1079] Step 9:

[1080] The server distributes the generated PDF flyer to registered users via email or messaging services. The input is the generated PDF file and the user's contact information, and the output is the flyer sent to the user. This allows users to receive the latest information on shopping mall deals in real time.

[1081] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1082] The present invention relates to a system that automatically collects product information provided by each store in a shopping mall, and creates and distributes the information in the form of a flyer while taking into consideration the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[1083] Data collection

[1084] server

[1085] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[1086] Drone / Ceiling Camera

[1087] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[1088] Shop owner (user)

[1089] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[1090] Data analysis

[1091] server

[1092] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[1093] Flyer generation

[1094] server

[1095] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[1096] Emotion Engine

[1097] server

[1098] The server is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's voice data and facial expression data and evaluates the user's emotions in real time. Based on this data, the emotion engine provides an interface and recommendation information that is adapted to the user's emotional state.

[1099] Information distribution

[1100] server

[1101] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The content of the interface is adjusted based on the user's emotional state by an emotion engine.

[1102] Specific examples

[1103] As an example, an embodiment in a shopping mall is shown below.

[1104] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[1105] 2. The captured image data is sent to the server in real time.

[1106] 3. The store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, they can record voice data such as "Apples are half price from 1:00 PM to 2:00 PM today."

[1107] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[1108] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[1109] 6. Before distributing the generated flyer to nearby residents, the server analyzes the user's emotional state using an emotion engine. For example, if the server detects that the user has recently been feeling stressed, it will provide an interface containing recommendations for soothing products and positive messages.

[1110] 7. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[1111] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently at low cost, thereby not only increasing the store's appeal to local consumers but also enabling the store to provide information according to the user's emotional state.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] server

[1115] The server sends a shooting command to the drone and ceiling camera at 9:00 a.m. This prepares each device to start shooting information for the designated store. The server sets the drone's flight route and shooting timing based on the shooting schedule and location information.

[1116] Step 2:

[1117] Drone / Ceiling Camera

[1118] The drones and ceiling cameras capture images of store products and pricing information according to a set schedule and location information. For example, the drone flies over a shopping district and captures images of each store's storefront, while the ceiling camera captures images of the product shelves inside the store. The captured image data is sent to a server in real time.

[1119] Step 3:

[1120] server

[1121] The server receives the captured image data and temporarily stores it in storage. It then uses image recognition algorithms to extract product names and price information from the image data. For example, it identifies product labels and price tags and converts them into text data. The extracted information is then stored in a database.

[1122] Step 4:

[1123] Shop owner (user)

[1124] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might input information like, "Apples are half price today from 1:00 p.m. to 2:00 p.m." The voice data is uploaded to the server in real time.

[1125] Step 5:

[1126] server

[1127] The server stores the received voice data and converts it into text using a speech recognition algorithm. The voice data is analyzed to extract special offers and limited-time sales information, which is then converted into text and stored in a database.

[1128] Step 6:

[1129] server

[1130] The server integrates product information, price information, and special offers stored in the database and automatically generates a flyer with "Today's Deals." The flyer is created in a visually easy-to-understand PDF format.

[1131] Step 7:

[1132] server

[1133] Before delivering the flyer, the server uses an emotion engine to evaluate the user's emotional state. It analyzes the user's voice and facial expression data to recognize their emotional state in real time. For example, if the user is feeling stressed through voice analysis, it will consider recommending relaxation products accordingly.

[1134] Step 8:

[1135] server

[1136] The generated flyer information is then adjusted based on the emotion engine's evaluation results. For example, designs and messages that stimulate positive emotions are added. The final flyer is then distributed to nearby residents at 11:00 AM.

[1137] Step 9:

[1138] server

[1139] The server retrieves registered user information and distributes flyers via messaging applications, such as LINE, to users, sending flyers in PDF format.

[1140] Step 10:

[1141] User

[1142] The user checks the received message and browses the flyer-style "Today's Deals." This allows the user to keep up with the latest deals from the shopping mall and check recommended products that are adapted to their emotional state.

[1143] In this way, the system of the present invention can automatically collect and analyze product information provided by each store in the shopping district, generate flyers, and provide information optimized based on the user's emotional state.

[1144] Example 2

[1145] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1146] Conventional systems have had difficulty collecting product information provided by stores in shopping districts and delivering that information appropriately, taking into account the user's emotional state. In particular, there was a need for a method to effectively aggregate product information and automate the delivery of information that reflects the user's emotional state. Furthermore, there was a need for a system that would allow store owners themselves to easily add information.

[1147] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing product and price information of stores in a commercial district, a means for recording voice data, a means for recognizing and extracting product names and price information from the photographed image data, a means for analyzing special offer information and limited-time sale information from the recorded voice data, a means for compiling the recognized, extracted, and analyzed information in an interface format, a means for recognizing the user's emotional state, a means for adjusting the generated interface based on the user's emotional state, and a means for delivering the interface to the user. This makes it possible to efficiently collect product information offered by each store in the shopping district and deliver information in the form of a flyer that takes the user's emotional state into consideration.

[1148] A "commercial district" is a specific area where multiple stores are located and where shopping and services are provided.

[1149] "Store" means a facility or place of business established to offer goods or services.

[1150] "Product and price information" is information about the names of products sold in the store and their selling prices.

[1151] "Means for photographing" refers to a method or apparatus for acquiring image data using a device such as a camera or drone.

[1152] "Voice data" refers to information collected through voice input, and is data that includes a particular human voice.

[1153] A "recording means" is a method or device for recording or capturing audio data.

[1154] "Image data" refers to visual information captured by a camera or drone stored in digital format.

[1155] "Means for recognition and extraction" refers to methods or devices that automatically detect and extract specific information from images or sounds.

[1156] "Special offer information" is information about products or services that are discounted below their regular price.

[1157] "Time sale information" is information about discount sales that take place during specific time periods.

[1158] "Means of analysis" are methods or devices for analyzing collected information and finding specific meaning or value.

[1159] An "interface" is a display format or means for providing information in a format that is visually easy for users to understand.

[1160] "User emotional state" refers to the user's emotional state, such as stress level or happiness.

[1161] A "means for recognizing" is a method or device for detecting and evaluating a user's emotions or state.

[1162] A "means for adjusting" is a method or device for optimizing information or display content based on the user's particular condition.

[1163] "Distribution means" refers to a method or device that uses communication means or a network to deliver information to users.

[1164] The present invention is a system that automatically collects product information offered by stores in a commercial area, and creates and distributes it in the form of a flyer, taking into account the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[1165] Data collection

[1166] server

[1167] The server manages the schedules of the drones and ceiling cameras to collect product information from each store in the commercial district. The server sends shooting instructions to the drones and ceiling cameras every day at 9:00 a.m. The devices operate according to these instructions and capture images of the store's products and price information.

[1168] Drone / Ceiling Camera

[1169] The drone and ceiling camera operate under instructions from the server. The drone photographs the store's exterior and merchandise, while the ceiling camera photographs the store's shelves. The captured image data is sent to the server in real time.

[1170] User

[1171] The store owner (user) uses a dedicated terminal application to input "Today's Deals" by voice. As an example of voice input, the terminal application records information such as "Apples are half price from 1pm to 2pm today," and the voice data is sent to the server.

[1172] Data analysis

[1173] server

[1174] The server analyzes the received image data using an image recognition algorithm to extract product name and price information. For example, the name of an "apple" and the price information "100 yen" are automatically extracted from the image data. The server also converts the received voice data into text using a voice recognition algorithm, and analyzes special offer information and limited time sale information. Through this analysis, special offer information such as "Apples are half price" is obtained as text.

[1175] Flyer generation

[1176] server

[1177] Based on the information stored in the database, the server automatically generates flyers with "Today's Deals" for the entire commercial district. The flyers are output as PDF files that display product images, names, prices, and special offers in a visually easy-to-understand format.

[1178] Emotion Engine

[1179] server

[1180] The server is equipped with an emotion engine that analyzes the user's voice and facial expression data to assess their emotional state. For example, if it determines from past data that the user has recently been feeling stressed, it will provide an interface containing products with a relaxing effect and positive messages.

[1181] Information distribution

[1182] server

[1183] The generated flyer is distributed to nearby residents via a messaging application (e.g., LINE). The server obtains registered user information, adjusts the interface content of the flyer based on the user's emotional state using an emotion engine, and distributes it at 12:00 a.m. Users can check the flyer from the received message and find out about the latest deals.

[1184] Specific examples

[1185] 1. The server sends a schedule to the drone and ceiling camera at 9:00 AM to "start filming."

[1186] 2. The drone flies over the commercial area, taking pictures of products and prices in each store and sending the images to the server. At the same time, a ceiling camera takes pictures of products inside the store and sends them to the server.

[1187] 3. The shopkeeper uses a dedicated terminal application to voice-input "Apples are half price." The voice data is uploaded to the server.

[1188] 4. The server analyzes the image data using an image recognition algorithm to extract product names and prices. It also converts the voice data into text using a voice recognition program to extract special offers and limited-time sales information.

[1189] 5. The server consolidates the analyzed information and automatically generates a flyer-format PDF file at 11:00 AM.

[1190] 6. The server uses an emotion engine to analyze past voice and facial expression data and adjust the flyer content based on the user's emotional state.

[1191] 7. At 12:00 AM, the server uses a messaging application to distribute the generated flyer to nearby residents. Users can then view the flyer in the message they receive and get the latest deals.

[1192] Example prompts for generative AI models

[1193] "Please explain in detail the specific steps involved in the system that automatically collects product information from each store in a commercial area, creates and distributes flyers in the form of flyers while taking into account the user's emotional state, including data collection, analysis of product information, flyer generation, and the function of the emotion engine."

[1194] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1195] Step 1: Prepare for data collection

[1196] server

[1197] The server sets a shooting schedule for the drone and ceiling camera at 9:00 AM every day and sends a shooting instruction. The server sets the schedule based on the current date and time as input for this instruction. As output, it generates an instruction for the drone and ceiling camera to start shooting. Based on this shooting instruction, the devices prepare to move on to the next step.

[1198] Step 2: Photograph your product information

[1199] Drone / Ceiling Camera

[1200] The drone and ceiling camera receive shooting instructions from the server and take pictures of products and price information in the commercial area. When receiving shooting instructions from the server as input, the drone takes pictures of the store's exterior and products in store, and the ceiling camera takes pictures of the product shelves inside the store. As output, the captured image data is generated and sent to the server in real time.

[1201] Specifically, the drone will fly over commercial areas along designated flight routes, taking high-resolution images of the exteriors of each store and the products on display, while the ceiling camera will continuously capture images of the shelves from a fixed position inside the store.

[1202] Step 3: Enter merchant information

[1203] User

[1204] The shop owner uses a dedicated terminal application to input "Today's Deals" by voice. As input, the shop owner enters voice data such as "Apples are half price today from 1 PM to 2 PM" into the terminal application. As output, this voice data is sent from the terminal to the server.

[1205] Specifically, the merchant launches the app, presses the voice input button on the screen, and then speaks the information. The voice is recorded by the app and instantly uploaded to the server.

[1206] Step 4: Collect image and audio data

[1207] server

[1208] The server receives and stores the captured image data and the audio data sent by the merchant. As input, the image data and audio data are sent to the server. As output, these data are stored in a temporary storage area within the server.

[1209] Specifically, as each piece of data is received by the server's data storage system, it is saved as a file in the appropriate directory, along with metadata such as the date and time of the photo and store information.

[1210] Step 5: Parse product information

[1211] server

[1212] The server uses an image recognition algorithm to extract product names and price information from the collected image data. It also uses a voice recognition algorithm to convert the merchant's voice data into text and analyze it. The saved image data and voice data are given as input. The analyzed text data, including product names, price information, and special offers, is generated as output.

[1213] Specifically, the server launches image recognition software to detect, for example, the product name "apple" and the price information "100 yen" from the image. Similarly, voice recognition software converts voice data such as "Apples are half price from 1:00 PM to 2:00 PM" into text and extracts it as special offer information.

[1214] Step 6: Store in the database

[1215] server

[1216] The server stores the parsed product information, price information, special offer information, and time sale information in a database. The parsed text data is used as input. The information stored in the database is generated as output.

[1217] Specifically, the server's database management system inserts the analysis results into the appropriate tables and associates each product data.

[1218] Step 7: Auto-generate flyers

[1219] server

[1220] The server automatically generates flyers containing "Today's Deals" for the entire commercial district based on the information stored in the database. As input, it accesses relevant information from the database. As output, it generates a visually easy-to-understand PDF file in flyer format.

[1221] Specifically, the server uses a proprietary layout algorithm to combine product images, names, and price information to generate a formatted PDF file.

[1222] Step 8: Analyzing the user's emotional state

[1223] server

[1224] The server uses an emotion engine to analyze the user's voice data and facial expression data and evaluate their emotional state. The user's past voice data and facial expression data are used as input. The output is evaluation data about the user's current emotional state.

[1225] Specifically, the server runs a machine learning model to calculate the user's stress level and happiness based on the input data.

[1226] Step 9: Prepare your flyer for distribution

[1227] server

[1228] The server adjusts the content of the flyer generated based on the results of the emotion engine and creates a message to be delivered. The flyer PDF and emotion rating data are used as input. The adjusted interface content message is generated as output.

[1229] Specifically, the server creates a message with content most suitable for the user in a specified format via the adaptive messaging system.

[1230] Step 10: Distribute flyers

[1231] server

[1232] The server distributes the generated flyer to nearby residents using a messaging application (e.g., LINE) at 12:00 AM. As input, the message with the adjusted interface content is used. As output, the flyer distributed through the messaging app is generated.

[1233] Specifically, the server uses a messaging API to send a message to all registered users, who can then click on the message to view the flyer.

[1234] (Application example 2)

[1235] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1236] In today's commercial environment, it is difficult to efficiently aggregate product information provided by stores and effectively distribute it to local consumers. Furthermore, there is a need to stimulate greater purchasing motivation by providing appropriate information tailored to consumers' emotional state. However, existing systems cannot meet these requirements and lack a means to quickly and effectively convey the latest special offers from shopping districts and individual stores to consumers. Therefore, a new system is needed to improve the consumer purchasing experience.

[1237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1238] In this invention, the server includes means for photographing product and price information in a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited time sale information from the recorded voice data, means for analyzing the user's emotional state, means for compiling the recognized, extracted, and analyzed information in an interface format adapted to the user's emotional state, and means for delivering the interface to the user. This makes it possible to quickly and effectively deliver the latest special offer information to consumers and to provide information tailored to the user's emotional state.

[1239] A "store" is a physical location for selling goods and services.

[1240] "Price information" is information about the selling price of a product.

[1241] "Photographing means" refers to a means of acquiring image data using a photographing device such as a camera or drone.

[1242] "Audio data" is digital data that contains a recording of a human voice.

[1243] "Product name" is the name given to a product that is distributed in the market.

[1244] "Recognition and extraction means" refers to the technical means for recognizing and extracting specific information from images and sounds.

[1245] "Special Offer Information" is information about special prices that are lower than the regular price.

[1246] "Time sale information" is information about sales that are only held during specific time periods.

[1247] An "emotional state" is the psychological state that a user is feeling at a given time.

[1248] "Analysis means" refers to means for analyzing and extracting information from digital data.

[1249] An "interface" is a display screen or operating means for exchanging information between a user and a computer system.

[1250] A "communications network" is an infrastructure for sending and receiving data between computers and devices.

[1251] A "delivery method" is a technique or method for delivering specific information to users.

[1252] The present invention relates to a system for efficiently collecting and analyzing product and price information from a store and delivering the information in an interface format that corresponds to the emotional state of a user. An embodiment of the system will be described in detail below.

[1253] Data collection

[1254] The server controls drones and ceiling cameras as a means of capturing images of store products and price information. These image capturing devices follow instructions from the server to collect image data including product and price information and send it to the server in real time. Store owners use a dedicated terminal application to record voice data. For example, they can input special offer information or limited-time sale information by voice, such as "Apples are half price from 1:00 PM to 2:00 PM today."

[1255] Data analysis

[1256] The server uses an image recognition algorithm (e.g., Python's OpenCV library) to analyze the received image data, thereby recognizing and extracting product names and prices, and storing them in a database. The voice data is converted into text using a speech recognition algorithm (e.g., Google Speech-to-Text API), and the server analyzes the special offers and limited-time sales information.

[1257] User sentiment analysis

[1258] Using the emotion engine, the server analyzes the user's voice data and facial expression data to evaluate their emotional state. The emotion engine uses a generative AI model (e.g., a Transformers model). This allows the server to recognize the user's emotional state, such as "feeling stressed" or "feeling happy," in real time.

[1259] Flyer generation and distribution

[1260] The server integrates the analyzed product information, price information, and special offer information to generate a flyer in an interface format that corresponds to the user's emotional state. The generated flyer is then distributed to users via a communication network. Specifically, it is distributed through a messaging application (e.g., LINE). In this way, consumers can obtain the latest special offer information in real time, increasing their motivation to purchase.

[1261] Examples and prompts

[1262] For example, at 9 a.m. in a shopping district, instructions to take product photos are sent from the server to drones and ceiling cameras. The store owner uses a dedicated terminal app to record voice data such as, "Apples are half price today from 1 p.m. to 2 p.m.." The data collected in this way is analyzed by the server and automatically generated as a flyer-style PDF file at 11 a.m.

[1263] The message the user receives will include the prompt:

[1264] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[1265] If the user has recently been identified as stressed:

[1266] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[1267] This will result in more personalized information and an improved consumer purchasing experience.

[1268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1269] Step 1:

[1270] Data collection: The server sends shooting instructions to the drones and ceiling cameras. Input includes the location information and shooting schedule of each store. Based on this, the drones and ceiling cameras take pictures of products and prices, and send the image data to the server in real time. The store owner uses a terminal application to input special offers and limited-time sales information by voice. This voice data is also sent to the server.

[1271] Step 2:

[1272] Image data analysis: The server receives the captured image data and uses an image recognition algorithm (Python's OpenCV library) to recognize and extract product name and price information. The input is the captured image data, and the output is text data containing product name and price information. Through this process, the information obtained from the image is converted into text format and stored in a database.

[1273] Step 3:

[1274] Voice data analysis: The server receives voice data from the merchant and converts it into text using a voice recognition algorithm (Google Speech-to-Text API). The input is voice data, and the output is text data containing special offers and time sale information. Through this process, the special offer information obtained from the voice is converted into text format and stored in a database.

[1275] Step 4:

[1276] Emotion analysis: The server receives the user's voice data and facial expression data and uses an emotion engine to analyze the user's emotional state. A generative AI model (Transformers model) evaluates the user's emotions from the input voice data and facial expression data. The input is the user's voice data and facial expression data, and the output is text data that indicates the user's emotional state. This processing allows the user's psychological state to be accurately understood.

[1277] Step 5:

[1278] Flyer generation: The server integrates the analyzed product information, price information, and special offer information to generate a flyer that corresponds to the user's emotional state. The input is product information, price information, special offer information, and the user's emotional state, and the output is an emotion-adaptive flyer. This flyer is formatted in a visually easy-to-understand format and contains information appropriate for the user. This process visualizes the purchasing information that is most suitable for the user.

[1279] Step 6:

[1280] Information distribution: The server distributes the generated flyer to the user over a communication network. Specifically, the flyer is sent via a messaging application. The input is the generated flyer data, and the output is the flyer received by the user. This process allows users to quickly receive the latest special offers and emotion-adaptive purchasing information.

[1281] Example prompt sentence:

[1282] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[1283] If the user has recently been identified as stressed:

[1284] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[1285] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1286] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1287] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1288] [Fourth embodiment]

[1289] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1290] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1291] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1292] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1293] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1294] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1295] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1296] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1297] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1298] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1299] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1300] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1301] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1302] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[1303] Data collection

[1304] server

[1305] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[1306] Drone / Ceiling Camera

[1307] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[1308] Shop owner (user)

[1309] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[1310] Data analysis

[1311] server

[1312] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[1313] Flyer generation

[1314] server

[1315] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[1316] Information distribution

[1317] server

[1318] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE).

[1319] Specific examples

[1320] As an example, an embodiment in a shopping mall is shown below.

[1321] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[1322] 2. The captured image data is sent to the server in real time.

[1323] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[1324] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[1325] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[1326] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[1327] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently and at low cost, thereby increasing its appeal to local consumers.

[1328] The processing flow will be explained below.

[1329] Step 1:

[1330] server

[1331] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m. It generates a shooting schedule and sets the drone's flight route based on the location information of the specified store. It also sets the ceiling camera to operate at a fixed time.

[1332] Step 2:

[1333] Drone / Ceiling Camera

[1334] The drone and ceiling camera follow instructions from the server to capture images of products and prices at designated times and locations, and the captured image data is sent to the server in real time.

[1335] Step 3:

[1336] server

[1337] The server stores the received image data and uses image recognition algorithms to extract product names and price information. Image analysis identifies product labels and price tags and stores them in a database in text format.

[1338] Step 4:

[1339] Shop owner (user)

[1340] Using a dedicated terminal application, the store owner enters "Today's Deals" by voice at 10:00 a.m. The terminal application records the voice data and uploads it to the server in real time.

[1341] Step 5:

[1342] server

[1343] The server stores the received voice data and converts it into text using a speech recognition algorithm. Through speech analysis, special offers and limited-time sales information are extracted and stored in a database in text format.

[1344] Step 6:

[1345] server

[1346] The server integrates product information, price information, and special offer information stored in the database, and automatically generates a flyer containing "Today's Deals" for the entire shopping district. It creates a visually easy-to-understand PDF file according to a template.

[1347] Step 7:

[1348] server

[1349] The server retrieves registered user information to distribute the generated flyer to nearby residents, and then distributes the flyer to the user at 11:00 AM using a messaging application (e.g., LINE).

[1350] Step 8:

[1351] User

[1352] Users can check the received message and view the flyer-style "Today's Deals," which allows them to keep up with the latest deals in the shopping district and shop efficiently.

[1353] In this way, the system of the present invention automatically collects and analyzes product information provided by each store in the shopping district, making it possible to efficiently provide consumers with the latest special price information.

[1354] Example 1

[1355] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1356] In the past, in order to quickly and efficiently communicate product information and special offers offered by each store in a shopping district to consumers, this was often done manually, which was time-consuming and costly. Furthermore, paper-based flyers had a large environmental impact and information updates were often delayed. Therefore, there was a growing need for an automated information provision system that could be updated in real time.

[1357] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1358] In this invention, the server includes means for photographing product and price information in stores using a drone and a ceiling camera and collecting the photographed data, means for recording voice data using a terminal application for speech input by the store owner, means for recognizing and extracting product names and price information from the collected image data using an image recognition algorithm, means for analyzing special offer information and limited-time sale information from the recorded voice data using a speech recognition algorithm, means for automatically generating a flyer-style interface using the recognized, extracted, and analyzed information, and means for distributing the generated flyer-style interface to registered users via a communication network. This enables the latest information on each store in the shopping district to be communicated to consumers quickly and efficiently.

[1359] A "drone" is an unmanned aerial vehicle that is flown by remote control or autonomous control and used to capture images of products and pricing information.

[1360] A "ceiling camera" is a fixed or movable camera device that is installed in a store and used to take images of shelves and merchandise in the store.

[1361] A "terminal application" is a device containing a software program used by a merchant to input voice data.

[1362] A "server" is a computer system that centrally manages data collection, analysis, storage, and distribution.

[1363] An "image recognition algorithm" is a software process for automatically extracting product name and price information from received image data.

[1364] "Speech recognition algorithm" means a software process that analyzes voice data provided by a merchant and converts it into textual information.

[1365] A "flyer-style interface" is a digital document that integrates product information and special offers and presents them in a visually easy-to-understand manner.

[1366] "Communications network" is a data transmission system used to distribute the generated flyer-style interface to registered users.

[1367] MODE FOR CARRYING OUT THE INVENTION

[1368] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments of the present invention will be described in detail below.

[1369] Data collection

[1370] server

[1371] The server controls the means of periodically collecting product information from each store in the shopping mall, including scheduling the drones and ceiling cameras, and sends shooting instructions to these devices.

[1372] Drone / Ceiling Camera

[1373] The drones and ceiling cameras follow instructions from the server and take images of products and price information for each store. The captured image data is sent to the server in real time. Specifically, drones equipped with high-resolution cameras fly over the shopping district, while ceiling cameras periodically take images of product shelves inside the stores. These devices instantly upload the data to the server via the network.

[1374] Shop owner (user)

[1375] The store owner uses a dedicated terminal application to provide "Today's Deals" by voice input. The voice data is automatically uploaded to the server. For example, "Apples are half price from 1pm to 2pm today."

[1376] Data analysis

[1377] server

[1378] The server stores the received image data and uses an image recognition algorithm to extract product name and price information. Specifically, it performs image analysis using libraries such as TensorFlow and OpenCV. It also uses a voice recognition algorithm (for example, a cloud-based voice recognition service) to convert the received voice data into text and analyzes the content. This allows it to extract information about special offers and limited-time sales. All analyzed data is stored in a database.

[1379] Flyer generation

[1380] server

[1381] Based on the information stored in the database, the server automatically generates flyers containing "Today's Deals" for the entire shopping district. This includes a program that integrates product information, price information, and special offers, and outputs the information in a format that is easy for users to understand visually. Specifically, the server uses tools such as Adobe InDesign Server to visually organize the integrated data and generate PDF files of the flyers.

[1382] Information distribution

[1383] server

[1384] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The distributed message includes a link to the flyer and a preview image, allowing users to easily check the contents.

[1385] Specific examples

[1386] As an example, an embodiment in a shopping mall is shown below.

[1387] 1. At 9:00 a.m., the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and photographs the products and prices of each store. At the same time, the ceiling camera photographs the product shelves inside the store.

[1388] 2. The captured image data is sent to the server in real time.

[1389] 3. At 10:00 a.m., the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the store owner might record voice data such as "Apples are half price from 1:00 p.m. to 2:00 p.m. today."

[1390] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[1391] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[1392] 6. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[1393] Prompt Sentence Examples

[1394] "This system collects product information from each store in the shopping district, automatically creates flyers, and distributes them to local residents. Products are photographed using drones and ceiling cameras, and the images are analyzed using TensorFlow and OpenCV. Store owners provide special offer information via voice input, which is converted into text using a speech recognition algorithm. This information is integrated and flyers are generated using Adobe InDesign Server. Finally, the flyers are distributed via the LINE Messaging API, making it easy for residents to access great deals."

[1395] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1396] Step 1: Prepare for data collection

[1397] server

[1398] The server begins preparing for data collection at 8:00 a.m. every day. It checks the schedule for each store and sets the operation schedule for the drones and ceiling cameras. The server then checks that each device is working properly. At this stage, the server confirms that all equipment is operating correctly and prepares for the next step.

[1399] Input: Shopping street store information, device operation status

[1400] Output: Shooting instructions and schedule settings for each device

[1401] Step 2: Photograph your product information

[1402] server

[1403] The server sends shooting instructions to the drone and ceiling camera at 9:00 a.m., specifically detailed instructions including shooting points and flight routes. The drone flies over the shopping district, photographing products and pricing information at each store. The ceiling camera periodically photographs the product shelves inside the store.

[1404] Drone / Ceiling Camera

[1405] The drone uses a high-resolution camera to capture images of products and prices in stores, while a ceiling camera captures images of the shelves inside the store. The captured image data is sent to a server in real time.

[1406] Input: Shooting instruction from the server

[1407] Output: Image data (product and price information)

[1408] Step 3: Recording audio data

[1409] Shop owner (user)

[1410] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might say, "Apples are half price today from 1:00 p.m. to 2:00 p.m." This information is then automatically uploaded to the server.

[1411] Input: Shopkeeper's voice data (discount information)

[1412] Output: Audio data file

[1413] Step 4: Analyzing the image data

[1414] server

[1415] The server stores the received image data and uses image recognition algorithms to extract product name and price information. Specifically, it analyzes the image using libraries such as TensorFlow and OpenCV and converts the product name and price information into digital data.

[1416] Input: Captured image data

[1417] Output: Extracted product and price information

[1418] Step 5: Analyzing the audio data

[1419] server

[1420] The server converts the received voice data into text using a voice recognition algorithm. Specifically, it uses a voice recognition service to analyze the voice data and converts the special offers and limited-time sale information into text data.

[1421] Input: Audio data file

[1422] Output: Extracted special offers and deals

[1423] Step 6: Integrate data and generate flyers

[1424] server

[1425] The server integrates the analyzed product information, price information, and special offer information, automatically generating a flyer-style interface, and using software such as Adobe InDesign Server to visually organize the integrated data and generate a PDF file of the flyer.

[1426] Input: Product information, price information, special offers

[1427] Output: Automatically generated flyer format PDF file

[1428] Step 7: Distribute flyers

[1429] server

[1430] The server obtains registered user information to distribute the generated flyers to nearby residents. The flyers are distributed via messages using the LINE Messaging API, etc. The distributed messages include a link to the flyer and a preview image, allowing users to easily check the contents.

[1431] Input: Automatically generated flyer format PDF file, user information

[1432] Output: Distributed flyers and messages

[1433] Through these steps, product information and special offer information collected from each store in the shopping district can be efficiently analyzed and integrated, and quickly delivered to consumers.

[1434] (Application example 1)

[1435] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1436] It is difficult to provide users with the latest store product information and special offers in real time. Furthermore, manually collecting information and generating flyers takes time and effort, making it inefficient. Furthermore, it is important that the information delivered is visually easy to understand and delivered in a timely manner. New methods are needed to solve these problems.

[1437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1438] In this invention, the server includes means for photographing product and price information from a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited-time sale information from the recorded voice data, means for compiling the recognized, extracted, and analyzed information in an interface format, means for generating this interface in a format viewable by users, and means for transmitting the generated interface via email or messaging service. This makes it possible to efficiently collect product and price information from a store in real time and provide it to users in a timely manner in a visually easy-to-understand format.

[1439] A "store" is a physical location for offering goods and services.

[1440] "Products" are goods and services sold to consumers.

[1441] "Price information" is information about the selling price of a product.

[1442] "Photographing means" refers to a device or system for capturing images of products.

[1443] "Audio data" means audio recorded in digital format.

[1444] A "recording means" is a device or system that records audio data.

[1445] "Photographed image data" refers to digital data of an image captured by a photographing device such as a camera.

[1446] The "product name" is a name that enables identification and recognition of the product.

[1447] "Means of recognition and extraction" refers to technologies and devices for identifying and extracting specific information from data.

[1448] "Special offer information" is information about products offered at special prices.

[1449] "Time sale information" is information about products for which special offers such as discounts are applied during specific time periods.

[1450] "Analytical means" are techniques and devices used to analyze data and interpret information.

[1451] An "interface format" is a display format that makes it easy for users to use information.

[1452] "Generative means" refers to the technology and devices used to create new information and forms based on data.

[1453] "Email" is a means of communication for sending and receiving messages over the Internet.

[1454] "Messaging Service" means a communication service for the real-time exchange of text and multimedia messages.

[1455] The present invention relates to a system for automatically collecting product information provided by each store in a shopping mall and creating the information in the form of a flyer. Specific embodiments will be described in detail below.

[1456] System Configuration

[1457] 1. Information gathering

[1458] How to take photos of store products and pricing information:

[1459] The server uses a camera or a smart eyeglasses or other imaging device to acquire product and price information from each store, thereby collecting image data of each store's products.

[1460] Methods for recording audio data:

[1461] Shop owners use a dedicated terminal application to input information about special offers and limited-time sales by voice. Voice input is done on a smartphone or a dedicated terminal, and the recorded voice data is automatically uploaded to a server.

[1462] 2. Data Analysis

[1463] Methods for recognizing and extracting product name and price information from image data:

[1464] The server uses image recognition algorithms such as Google Cloud Vision API to extract product name and price information from the captured image data.

[1465] How to analyze special offers and time-limited sales information from recorded audio data:

[1466] The server uses speech recognition algorithms such as the Google Cloud Speech-to-Text API to convert the recorded audio data into text, and then analyzes the content to extract special offers and limited-time sales information.

[1467] 3. Flyer generation

[1468] Means of organizing the recognized, extracted and analyzed information in an interface format:

[1469] The server integrates the information stored in the database and generates flyers in a format that is easy for users to understand visually. It uses libraries such as FPDF to automatically generate flyers in PDF format.

[1470] A means of producing the generated interface in a user-viewable format:

[1471] The flyers are printed in a format that can be viewed on smartphones or smart glasses.

[1472] 4. Information distribution

[1473] How to send the generated interface via email or messaging services:

[1474] The server then distributes the generated flyer to users via email or messaging services (e.g., LINE), allowing users to receive the latest special offers and time-limited sales information in real time.

[1475] Examples:

[1476] For example, an example implementation in a shopping mall is shown below. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera to capture images of products and prices at each store. At the same time, the ceiling camera captures images of the store's shelves. At 10:00 AM, the store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, the voice data could include "Apples are half price from 1:00 PM to 2:00 PM today." The server then integrates the recognized and extracted information and automatically generates a PDF file in flyer format at 11:00 AM. At 12:00 PM, the server distributes the generated flyer to registered users using a messaging application.

[1477] Example prompt sentence:

[1478] 1. Take a photo of the product.

[1479] 2. Enter product information and special offers using your voice.

[1480] 3. Perform image and voice recognition and integrate the extracted information into a flyer format.

[1481] 4. Review the generated flyer and make any necessary corrections.

[1482] 5. The completed flyer will be sent to registered users via email or LINE.

[1483] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1484] Step 1:

[1485] The server sends a shooting instruction to the drone and ceiling camera at 9:00 AM. The input is the schedule data for the previous day, and the output is a command to start shooting. The server sends this command to the drone and camera, causing them to take pictures of the store's products and price information.

[1486] Step 2:

[1487] The terminal uploads the captured image data to the server in real time. The input is the image data captured by the drone and ceiling camera, and the output is the image data uploaded to the server. The terminal then completes the collection of image data.

[1488] Step 3:

[1489] The server sends a notification to the merchant for voice input at 10:00 AM. The input is the preset notification schedule and the output is sending the notification to the merchant. The server then prompts the merchant to input voice data.

[1490] Step 4:

[1491] The merchant uses a dedicated terminal application to input "Today's Deals" by voice. The input is the merchant's voice, and the output is voice data stored on the terminal. The merchant then uploads this voice data to the terminal.

[1492] Step 5:

[1493] The terminal uploads the recorded voice data to the server in real time. The input is the voice data recorded by the merchant, and the output is the voice data uploaded to the server.

[1494] Step 6:

[1495] The server uses the Google Cloud Vision API to recognize and extract product names and price information from image data. The input is the uploaded image data, and the output is text data with the product names and price information extracted. The server stores this data in a database.

[1496] Step 7:

[1497] The server uses the Google Cloud Speech-to-Text API to convert the recorded voice data into text and analyze the special offer and time sale information. The input is the uploaded voice data, and the output is the analyzed text data of the special offer and time sale information. The server stores this data in a database.

[1498] Step 8:

[1499] The server integrates product information, price information, special offer information, and time sale information stored in the database, and automatically generates a PDF file in flyer format using the FPDF library. The input is the information stored in the database, and the output is the generated PDF flyer.

[1500] Step 9:

[1501] The server distributes the generated PDF flyer to registered users via email or messaging services. The input is the generated PDF file and the user's contact information, and the output is the flyer sent to the user. This allows users to receive the latest information on shopping mall deals in real time.

[1502] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1503] The present invention relates to a system that automatically collects product information provided by each store in a shopping mall, and creates and distributes the information in the form of a flyer while taking into consideration the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[1504] Data collection

[1505] server

[1506] The server controls the periodic collection of product information from each store in the shopping mall, manages the schedules for the drones and ceiling cameras, and sends shooting instructions to these devices.

[1507] Drone / Ceiling Camera

[1508] Drones and ceiling cameras follow instructions from the server to capture images of products and prices in stores, and the captured image data is sent to the server in real time.

[1509] Shop owner (user)

[1510] Using a dedicated terminal application, shop owners can input "Today's Deals" by voice, and the voice data is automatically uploaded to the server.

[1511] Data analysis

[1512] server

[1513] The server stores the received image data and uses an image recognition algorithm to extract product names and prices. It also converts the received voice data into text using a voice recognition algorithm, analyzes the content, and extracts special offers and limited-time sales information. All of this data is stored in a database.

[1514] Flyer generation

[1515] server

[1516] Based on the information stored in the database, a flyer containing "Today's Deals" for the entire shopping district is automatically generated. The program for this purpose integrates product information, price information, and special offers, and outputs them in a format that is easy for users to understand visually.

[1517] Emotion Engine

[1518] server

[1519] The server is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's voice data and facial expression data and evaluates the user's emotions in real time. Based on this data, the emotion engine provides an interface and recommendation information that is adapted to the user's emotional state.

[1520] Information distribution

[1521] server

[1522] The generated flyers are distributed to nearby residents. The server obtains registered user information and distributes the flyers via a messaging application (e.g., LINE). The content of the interface is adjusted based on the user's emotional state by an emotion engine.

[1523] Specific examples

[1524] As an example, an embodiment in a shopping mall is shown below.

[1525] 1. At 9:00 AM, the server sends a shooting command to the drone and ceiling camera. The drone flies over the shopping district and takes pictures of the products and prices in each store. At the same time, the ceiling camera takes pictures of the product shelves inside the store.

[1526] 2. The captured image data is sent to the server in real time.

[1527] 3. The store owner uses a dedicated terminal application to input "Today's Deals" by voice. For example, they can record voice data such as "Apples are half price from 1:00 PM to 2:00 PM today."

[1528] 4. The server analyzes the received image data using an image recognition algorithm to extract product name and price information. It also converts the voice data into text using a voice recognition algorithm to extract special offers and limited-time sale information.

[1529] 5. The server consolidates this information and automatically generates a flyer-format PDF file at 11:00 AM.

[1530] 6. Before distributing the generated flyer to nearby residents, the server analyzes the user's emotional state using an emotion engine. For example, if the server detects that the user has recently been feeling stressed, it will provide an interface containing recommendations for soothing products and positive messages.

[1531] 7. At 12:00 PM, the server sends the generated flyer to registered users via a messaging application. Users can check the flyer and get information about special offers from the message they receive.

[1532] The present invention enables each store in a shopping district to provide consumers with the latest bargain information efficiently at low cost, thereby not only increasing the store's appeal to local consumers but also enabling the store to provide information according to the user's emotional state.

[1533] The processing flow will be explained below.

[1534] Step 1:

[1535] server

[1536] The server sends a shooting command to the drone and ceiling camera at 9:00 a.m. This prepares each device to start shooting information for the designated store. The server sets the drone's flight route and shooting timing based on the shooting schedule and location information.

[1537] Step 2:

[1538] Drone / Ceiling Camera

[1539] The drones and ceiling cameras capture images of store products and pricing information according to a set schedule and location information. For example, the drone flies over a shopping district and captures images of each store's storefront, while the ceiling camera captures images of the product shelves inside the store. The captured image data is sent to a server in real time.

[1540] Step 3:

[1541] server

[1542] The server receives the captured image data and temporarily stores it in storage. It then uses image recognition algorithms to extract product names and price information from the image data. For example, it identifies product labels and price tags and converts them into text data. The extracted information is then stored in a database.

[1543] Step 4:

[1544] Shop owner (user)

[1545] At 10:00 a.m., the store owner uses a dedicated terminal application to voice-input "Today's Deals." For example, they might input information like, "Apples are half price today from 1:00 p.m. to 2:00 p.m." The voice data is uploaded to the server in real time.

[1546] Step 5:

[1547] server

[1548] The server stores the received voice data and converts it into text using a speech recognition algorithm. The voice data is analyzed to extract special offers and limited-time sales information, which is then converted into text and stored in a database.

[1549] Step 6:

[1550] server

[1551] The server integrates product information, price information, and special offers stored in the database and automatically generates a flyer with "Today's Deals." The flyer is created in a visually easy-to-understand PDF format.

[1552] Step 7:

[1553] server

[1554] Before delivering the flyer, the server uses an emotion engine to evaluate the user's emotional state. It analyzes the user's voice and facial expression data to recognize their emotional state in real time. For example, if the user is feeling stressed through voice analysis, it will consider recommending relaxation products accordingly.

[1555] Step 8:

[1556] server

[1557] The generated flyer information is then adjusted based on the emotion engine's evaluation results. For example, designs and messages that stimulate positive emotions are added. The final flyer is then distributed to nearby residents at 11:00 AM.

[1558] Step 9:

[1559] server

[1560] The server retrieves registered user information and distributes flyers via messaging applications, such as LINE, to users, sending flyers in PDF format.

[1561] Step 10:

[1562] User

[1563] The user checks the received message and browses the flyer-style "Today's Deals." This allows the user to keep up with the latest deals from the shopping mall and check recommended products that are adapted to their emotional state.

[1564] In this way, the system of the present invention can automatically collect and analyze product information provided by each store in the shopping district, generate flyers, and provide information optimized based on the user's emotional state.

[1565] Example 2

[1566] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1567] Conventional systems have had difficulty collecting product information provided by stores in shopping districts and delivering that information appropriately, taking into account the user's emotional state. In particular, there was a need for a method to effectively aggregate product information and automate the delivery of information that reflects the user's emotional state. Furthermore, there was a need for a system that would allow store owners themselves to easily add information.

[1568] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing product and price information of stores in a commercial district, a means for recording voice data, a means for recognizing and extracting product names and price information from the photographed image data, a means for analyzing special offer information and limited-time sale information from the recorded voice data, a means for compiling the recognized, extracted, and analyzed information in an interface format, a means for recognizing the user's emotional state, a means for adjusting the generated interface based on the user's emotional state, and a means for delivering the interface to the user. This makes it possible to efficiently collect product information offered by each store in the shopping district and deliver information in the form of a flyer that takes the user's emotional state into consideration.

[1569] A "commercial district" is a specific area where multiple stores are located and where shopping and services are provided.

[1570] "Store" means a facility or place of business established to offer goods or services.

[1571] "Product and price information" is information about the names of products sold in the store and their selling prices.

[1572] "Means for photographing" refers to a method or apparatus for acquiring image data using a device such as a camera or drone.

[1573] "Voice data" refers to information collected through voice input, and is data that includes a particular human voice.

[1574] A "recording means" is a method or device for recording or capturing audio data.

[1575] "Image data" refers to visual information captured by a camera or drone stored in digital format.

[1576] "Means for recognition and extraction" refers to methods or devices that automatically detect and extract specific information from images or sounds.

[1577] "Special offer information" is information about products or services that are discounted below their regular price.

[1578] "Time sale information" is information about discount sales that take place during specific time periods.

[1579] "Means of analysis" are methods or devices for analyzing collected information and finding specific meaning or value.

[1580] An "interface" is a display format or means for providing information in a format that is visually easy for users to understand.

[1581] "User emotional state" refers to the user's emotional state, such as stress level or happiness.

[1582] A "means for recognizing" is a method or device for detecting and evaluating a user's emotions or state.

[1583] A "means for adjusting" is a method or device for optimizing information or display content based on the user's particular condition.

[1584] "Distribution means" refers to a method or device that uses communication means or a network to deliver information to users.

[1585] The present invention is a system that automatically collects product information offered by stores in a commercial area, and creates and distributes it in the form of a flyer, taking into account the emotional state of the user. Specific embodiments of the present invention are described in detail below.

[1586] Data collection

[1587] server

[1588] The server manages the schedules of the drones and ceiling cameras to collect product information from each store in the commercial district. The server sends shooting instructions to the drones and ceiling cameras every day at 9:00 a.m. The devices operate according to these instructions and capture images of the store's products and price information.

[1589] Drone / Ceiling Camera

[1590] The drone and ceiling camera operate under instructions from the server. The drone photographs the store's exterior and merchandise, while the ceiling camera photographs the store's shelves. The captured image data is sent to the server in real time.

[1591] User

[1592] The store owner (user) uses a dedicated terminal application to input "Today's Deals" by voice. As an example of voice input, the terminal application records information such as "Apples are half price from 1pm to 2pm today," and the voice data is sent to the server.

[1593] Data analysis

[1594] server

[1595] The server analyzes the received image data using an image recognition algorithm to extract product name and price information. For example, the name of an "apple" and the price information "100 yen" are automatically extracted from the image data. The server also converts the received voice data into text using a voice recognition algorithm, and analyzes special offer information and limited time sale information. Through this analysis, special offer information such as "Apples are half price" is obtained as text.

[1596] Flyer generation

[1597] server

[1598] Based on the information stored in the database, the server automatically generates flyers with "Today's Deals" for the entire commercial district. The flyers are output as PDF files that display product images, names, prices, and special offers in a visually easy-to-understand format.

[1599] Emotion Engine

[1600] server

[1601] The server is equipped with an emotion engine that analyzes the user's voice and facial expression data to assess their emotional state. For example, if it determines from past data that the user has recently been feeling stressed, it will provide an interface containing products with a relaxing effect and positive messages.

[1602] Information distribution

[1603] server

[1604] The generated flyer is distributed to nearby residents via a messaging application (e.g., LINE). The server obtains registered user information, adjusts the interface content of the flyer based on the user's emotional state using an emotion engine, and distributes it at 12:00 a.m. Users can check the flyer from the received message and find out about the latest deals.

[1605] Specific examples

[1606] 1. The server sends a schedule to the drone and ceiling camera at 9:00 AM to "start filming."

[1607] 2. The drone flies over the commercial area, taking pictures of products and prices in each store and sending the images to the server. At the same time, a ceiling camera takes pictures of products inside the store and sends them to the server.

[1608] 3. The shopkeeper uses a dedicated terminal application to voice-input "Apples are half price." The voice data is uploaded to the server.

[1609] 4. The server analyzes the image data using an image recognition algorithm to extract product names and prices. It also converts the voice data into text using a voice recognition program to extract special offers and limited-time sales information.

[1610] 5. The server consolidates the analyzed information and automatically generates a flyer-format PDF file at 11:00 AM.

[1611] 6. The server uses an emotion engine to analyze past voice and facial expression data and adjust the flyer content based on the user's emotional state.

[1612] 7. At 12:00 AM, the server uses a messaging application to distribute the generated flyer to nearby residents. Users can then view the flyer in the message they receive and get the latest deals.

[1613] Example prompts for generative AI models

[1614] "Please explain in detail the specific steps involved in the system that automatically collects product information from each store in a commercial area, creates and distributes flyers in the form of flyers while taking into account the user's emotional state, including data collection, analysis of product information, flyer generation, and the function of the emotion engine."

[1615] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1616] Step 1: Prepare for data collection

[1617] server

[1618] The server sets a shooting schedule for the drone and ceiling camera at 9:00 AM every day and sends a shooting instruction. The server sets the schedule based on the current date and time as input for this instruction. As output, it generates an instruction for the drone and ceiling camera to start shooting. Based on this shooting instruction, the devices prepare to move on to the next step.

[1619] Step 2: Photograph your product information

[1620] Drone / Ceiling Camera

[1621] The drone and ceiling camera receive shooting instructions from the server and take pictures of products and price information in the commercial area. When receiving shooting instructions from the server as input, the drone takes pictures of the store's exterior and products in store, and the ceiling camera takes pictures of the product shelves inside the store. As output, the captured image data is generated and sent to the server in real time.

[1622] Specifically, the drone will fly over commercial areas along designated flight routes, taking high-resolution images of the exteriors of each store and the products on display, while the ceiling camera will continuously capture images of the shelves from a fixed position inside the store.

[1623] Step 3: Enter merchant information

[1624] User

[1625] The shop owner uses a dedicated terminal application to input "Today's Deals" by voice. As input, the shop owner enters voice data such as "Apples are half price today from 1 PM to 2 PM" into the terminal application. As output, this voice data is sent from the terminal to the server.

[1626] Specifically, the merchant launches the app, presses the voice input button on the screen, and then speaks the information. The voice is recorded by the app and instantly uploaded to the server.

[1627] Step 4: Collect image and audio data

[1628] server

[1629] The server receives and stores the captured image data and the audio data sent by the merchant. As input, the image data and audio data are sent to the server. As output, these data are stored in a temporary storage area within the server.

[1630] Specifically, as each piece of data is received by the server's data storage system, it is saved as a file in the appropriate directory, along with metadata such as the date and time of the photo and store information.

[1631] Step 5: Parse product information

[1632] server

[1633] The server uses an image recognition algorithm to extract product names and price information from the collected image data. It also uses a voice recognition algorithm to convert the merchant's voice data into text and analyze it. The saved image data and voice data are given as input. The analyzed text data, including product names, price information, and special offers, is generated as output.

[1634] Specifically, the server launches image recognition software to detect, for example, the product name "apple" and the price information "100 yen" from the image. Similarly, voice recognition software converts voice data such as "Apples are half price from 1:00 PM to 2:00 PM" into text and extracts it as special offer information.

[1635] Step 6: Store in the database

[1636] server

[1637] The server stores the parsed product information, price information, special offer information, and time sale information in a database. The parsed text data is used as input. The information stored in the database is generated as output.

[1638] Specifically, the server's database management system inserts the analysis results into the appropriate tables and associates each product data.

[1639] Step 7: Auto-generate flyers

[1640] server

[1641] The server automatically generates flyers containing "Today's Deals" for the entire commercial district based on the information stored in the database. As input, it accesses relevant information from the database. As output, it generates a visually easy-to-understand PDF file in flyer format.

[1642] Specifically, the server uses a proprietary layout algorithm to combine product images, names, and price information to generate a formatted PDF file.

[1643] Step 8: Analyzing the user's emotional state

[1644] server

[1645] The server uses an emotion engine to analyze the user's voice data and facial expression data and evaluate their emotional state. The user's past voice data and facial expression data are used as input. The output is evaluation data about the user's current emotional state.

[1646] Specifically, the server runs a machine learning model to calculate the user's stress level and happiness based on the input data.

[1647] Step 9: Prepare your flyer for distribution

[1648] server

[1649] The server adjusts the content of the flyer generated based on the results of the emotion engine and creates a message to be delivered. The flyer PDF and emotion rating data are used as input. The adjusted interface content message is generated as output.

[1650] Specifically, the server creates a message with content most suitable for the user in a specified format via the adaptive messaging system.

[1651] Step 10: Distribute flyers

[1652] server

[1653] The server distributes the generated flyer to nearby residents using a messaging application (e.g., LINE) at 12:00 AM. As input, the message with the adjusted interface content is used. As output, the flyer distributed through the messaging app is generated.

[1654] Specifically, the server uses a messaging API to send a message to all registered users, who can then click on the message to view the flyer.

[1655] (Application example 2)

[1656] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1657] In today's commercial environment, it is difficult to efficiently aggregate product information provided by stores and effectively distribute it to local consumers. Furthermore, there is a need to stimulate greater purchasing motivation by providing appropriate information tailored to consumers' emotional state. However, existing systems cannot meet these requirements and lack a means to quickly and effectively convey the latest special offers from shopping districts and individual stores to consumers. Therefore, a new system is needed to improve the consumer purchasing experience.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1659] In this invention, the server includes means for photographing product and price information in a store, means for recording voice data, means for recognizing and extracting product names and price information from the photographed image data, means for analyzing special offer information and limited time sale information from the recorded voice data, means for analyzing the user's emotional state, means for compiling the recognized, extracted, and analyzed information in an interface format adapted to the user's emotional state, and means for delivering the interface to the user. This makes it possible to quickly and effectively deliver the latest special offer information to consumers and to provide information tailored to the user's emotional state.

[1660] A "store" is a physical location for selling goods and services.

[1661] "Price information" is information about the selling price of a product.

[1662] "Photographing means" refers to a means of acquiring image data using a photographing device such as a camera or drone.

[1663] "Audio data" is digital data that contains a recording of a human voice.

[1664] "Product name" is the name given to a product that is distributed in the market.

[1665] "Recognition and extraction means" refers to the technical means for recognizing and extracting specific information from images and sounds.

[1666] "Special Offer Information" is information about special prices that are lower than the regular price.

[1667] "Time sale information" is information about sales that are only held during specific time periods.

[1668] An "emotional state" is the psychological state that a user is feeling at a given time.

[1669] "Analysis means" refers to means for analyzing and extracting information from digital data.

[1670] An "interface" is a display screen or operating means for exchanging information between a user and a computer system.

[1671] A "communications network" is an infrastructure for sending and receiving data between computers and devices.

[1672] A "delivery method" is a technique or method for delivering specific information to users.

[1673] The present invention relates to a system for efficiently collecting and analyzing product and price information from a store and delivering the information in an interface format that corresponds to the emotional state of a user. An embodiment of the system will be described in detail below.

[1674] Data collection

[1675] The server controls drones and ceiling cameras as a means of capturing images of store products and price information. These image capturing devices follow instructions from the server to collect image data including product and price information and send it to the server in real time. Store owners use a dedicated terminal application to record voice data. For example, they can input special offer information or limited-time sale information by voice, such as "Apples are half price from 1:00 PM to 2:00 PM today."

[1676] Data analysis

[1677] The server uses an image recognition algorithm (e.g., Python's OpenCV library) to analyze the received image data, thereby recognizing and extracting product names and prices, and storing them in a database. The voice data is converted into text using a speech recognition algorithm (e.g., Google Speech-to-Text API), and the server analyzes the special offers and limited-time sales information.

[1678] User sentiment analysis

[1679] Using the emotion engine, the server analyzes the user's voice data and facial expression data to evaluate their emotional state. The emotion engine uses a generative AI model (e.g., a Transformers model). This allows the server to recognize the user's emotional state, such as "feeling stressed" or "feeling happy," in real time.

[1680] Flyer generation and distribution

[1681] The server integrates the analyzed product information, price information, and special offer information to generate a flyer in an interface format that corresponds to the user's emotional state. The generated flyer is then distributed to users via a communication network. Specifically, it is distributed through a messaging application (e.g., LINE). In this way, consumers can obtain the latest special offer information in real time, increasing their motivation to purchase.

[1682] Examples and prompts

[1683] For example, at 9 a.m. in a shopping district, instructions to take product photos are sent from the server to drones and ceiling cameras. The store owner uses a dedicated terminal app to record voice data such as, "Apples are half price today from 1 p.m. to 2 p.m.." The data collected in this way is analyzed by the server and automatically generated as a flyer-style PDF file at 11 a.m.

[1684] The message the user receives will include the prompt:

[1685] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[1686] If the user has recently been identified as stressed:

[1687] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[1688] This will result in more personalized information and an improved consumer purchasing experience.

[1689] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1690] Step 1:

[1691] Data collection: The server sends shooting instructions to the drones and ceiling cameras. Input includes the location information and shooting schedule of each store. Based on this, the drones and ceiling cameras take pictures of products and prices, and send the image data to the server in real time. The store owner uses a terminal application to input special offers and limited-time sales information by voice. This voice data is also sent to the server.

[1692] Step 2:

[1693] Image data analysis: The server receives the captured image data and uses an image recognition algorithm (Python's OpenCV library) to recognize and extract product name and price information. The input is the captured image data, and the output is text data containing product name and price information. Through this process, the information obtained from the image is converted into text format and stored in a database.

[1694] Step 3:

[1695] Voice data analysis: The server receives voice data from the merchant and converts it into text using a voice recognition algorithm (Google Speech-to-Text API). The input is voice data, and the output is text data containing special offers and time sale information. Through this process, the special offer information obtained from the voice is converted into text format and stored in a database.

[1696] Step 4:

[1697] Emotion analysis: The server receives the user's voice data and facial expression data and uses an emotion engine to analyze the user's emotional state. A generative AI model (Transformers model) evaluates the user's emotions from the input voice data and facial expression data. The input is the user's voice data and facial expression data, and the output is text data that indicates the user's emotional state. This processing allows the user's psychological state to be accurately understood.

[1698] Step 5:

[1699] Flyer generation: The server integrates the analyzed product information, price information, and special offer information to generate a flyer that corresponds to the user's emotional state. The input is product information, price information, special offer information, and the user's emotional state, and the output is an emotion-adaptive flyer. This flyer is formatted in a visually easy-to-understand format and contains information appropriate for the user. This process visualizes the purchasing information that is most suitable for the user.

[1700] Step 6:

[1701] Information distribution: The server distributes the generated flyer to the user over a communication network. Specifically, the flyer is sent via a messaging application. The input is the generated flyer data, and the output is the flyer received by the user. This process allows users to quickly receive the latest special offers and emotion-adaptive purchasing information.

[1702] Example prompt sentence:

[1703] "Apples are half price from 1pm to 2pm today. Don't miss out!"

[1704] If the user has recently been identified as stressed:

[1705] "It seems like you've been stressed lately. I recommend some delicious sweet apples as a soothing product!"

[1706] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1707] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1708] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1709] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1710] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1711] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1712] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1713] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1714] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1715] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1716] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1717] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1718] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1719] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1720] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1721] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1722] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1723] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1724] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1725] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1726] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1727] The following is further disclosed regarding the above embodiment.

[1728] (Claim 1)

[1729] A means for taking pictures of products and price information in a store;

[1730] a means for recording audio data;

[1731] A means for recognizing and extracting product name and price information from the captured image data;

[1732] A means for analyzing special offer information and time sale information from the recorded voice data;

[1733] A means for compiling the recognized, extracted and analyzed information in an interface format;

[1734] a means for delivering the interface to a user;

[1735] A system including:

[1736] (Claim 2)

[1737] 2. The system of claim 1, wherein the means for recording voice data includes a terminal application for the merchant to input voice data.

[1738] (Claim 3)

[1739] 2. The system of claim 1, wherein the means for delivering the interface to the user is performed via a communications network.

[1740] (Claim 4)

[1741] 2. The system according to claim 1, wherein the means for recognizing and extracting product name and price information from the captured image data uses an image recognition algorithm.

[1742] (Claim 5)

[1743] 2. The system of claim 1, wherein the means for analyzing the special offer information and limited time sale information from the recorded voice data uses a voice recognition algorithm.

[1744] (Claim 6)

[1745] 10. The system of claim 1, wherein the means for compiling in an interface format includes means for automatically generating a document in a flyer format.

[1746] (Claim 7)

[1747] 10. The system of claim 1, wherein the means for delivering the interface to the user is via a messaging application.

[1748] (Claim 8)

[1749] 10. The system of claim 1, wherein the means for photographing the store's product and price information includes a drone or a ceiling camera.

[1750] (Claim 9)

[1751] 10. The system of claim 1, wherein the means for compiling in an interface format includes means for storing the generated flyer information in a database.

[1752] "Example 1"

[1753] (Claim 1)

[1754] A means for capturing images of products and price information in stores using drones and ceiling cameras and collecting the images as data;

[1755] a means for recording voice data using a terminal application for voice input by the store owner;

[1756] A means for recognizing and extracting product name and price information from the collected image data using an image recognition algorithm;

[1757] A means for analyzing the recorded voice data using a voice recognition algorithm to provide special offer information and limited time sale information;

[1758] A means for automatically generating a flyer-style interface using the recognized, extracted, and analyzed information;

[1759] means for distributing the generated flyer-style interface to registered users via a communication network;

[1760] A system including:

[1761] (Claim 2)

[1762] 10. The system of claim 1, further comprising a terminal application for merchant voice input.

[1763] (Claim 3)

[1764] 10. The system of claim 1, further comprising means for delivering the interface to a user over a communications network.

[1765] "Application Example 1"

[1766] (Claim 1)

[1767] A means for taking pictures of products and price information in a store;

[1768] a means for recording audio data;

[1769] A means for recognizing and extracting product name and price information from the captured image data;

[1770] A means for analyzing special offer information and time sale information from the recorded voice data;

[1771] A means for compiling the recognized, extracted and analyzed information in an interface format;

[1772] a means for generating the interface in a user-viewable format; and

[1773] means for transmitting the generated interface via email or messaging service;

[1774] A system including:

[1775] (Claim 2)

[1776] 2. The system of claim 1, wherein the means for recording voice data includes a terminal application for merchant voice input.

[1777] (Claim 3)

[1778] 10. The system of claim 1, wherein the means for delivering the interface to the user is performed over a communications network.

[1779] "Example 2: Combining Emotion Engines"

[1780] (Claim 1)

[1781] a means for photographing merchandise and price information of stores in the commercial area;

[1782] a means for recording audio data;

[1783] A means for recognizing and extracting product name and price information from the captured image data;

[1784] A means for analyzing special offer information and time sale information from the recorded audio data;

[1785] A means for compiling the recognized, extracted and analyzed information in an interface format;

[1786] means for recognizing the emotional state of a user;

[1787] means for adjusting the generated interface based on the emotional state of the user;

[1788] a means for delivering the interface to a user;

[1789] A system including:

[1790] (Claim 2)

[1791] 2. The system of claim 1, wherein the means for recording voice data includes a terminal application for merchant voice input.

[1792] (Claim 3)

[1793] 10. The system of claim 1, wherein the means for delivering the interface to the user is performed over a communications network.

[1794] "Application example 2 when combining emotion engines"

[1795] (Claim 1)

[1796] A means for taking pictures of products and price information in a store;

[1797] a means for recording audio data;

[1798] A means for recognizing and extracting product name and price information from the captured image data;

[1799] A means for analyzing special offer information and time sale information from the recorded voice data;

[1800] means for analyzing the emotional state of a user;

[1801] A means for organizing the recognized, extracted and analyzed information in an interface format adapted to the user's emotional state;

[1802] a means for delivering the interface to a user;

[1803] A system including:

[1804] (Claim 2)

[1805] 2. The system of claim 1, wherein the means for recording voice data includes a terminal application for merchant voice input.

[1806] (Claim 3)

[1807] 10. The system of claim 1, wherein the means for delivering the interface to the user is performed over a communications network. [Explanation of symbols]

[1808] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for taking pictures of products and price information in a store; a means for recording audio data; A means for recognizing and extracting product name and price information from the captured image data; A means for analyzing special offer information and time sale information from the recorded voice data; a means for compiling the recognized, extracted and analyzed information in an interface format; a means for delivering the interface to a user; A system including:

2. 2. The system of claim 1, wherein the means for recording voice data includes a terminal application for allowing the merchant to input voice data.

3. 2. The system of claim 1, wherein the means for delivering the interface to the user is performed via a communications network.

4. 2. The system according to claim 1, wherein the means for recognizing and extracting product name and price information from the photographed image data uses an image recognition algorithm.

5. 2. The system of claim 1, wherein the means for analyzing the special offer and limited time sale information from the recorded voice data uses a voice recognition algorithm.

6. 2. The system of claim 1, wherein the means for compiling in an interface format includes means for automatically generating a document in a flyer format.

7. 10. The system of claim 1, wherein the means for delivering the interface to the user is via a messaging application.

8. 10. The system according to claim 1, wherein the means for photographing the store's product and price information includes a drone or a ceiling camera.

9. 2. The system of claim 1, wherein the means for compiling in an interface format includes means for storing the generated flyer information in a database.

Images