system

Abstract

The system according to this embodiment aims to enable users to stay informed about point campaigns and special offers in real time and to efficiently earn points. [Solution] The system according to the embodiment comprises a recognition unit, a notification unit, a voice recognition unit, a voice notification unit, and a suggestion unit. The recognition unit recognizes products and advertisements captured by the camera. The notification unit notifies the user of point campaigns based on the products and advertisements recognized by the recognition unit. The voice recognition unit recognizes in-store announcements and conversations. The voice notification unit notifies the user of special sale information and campaigns recognized by the voice recognition unit. The suggestion unit uses GPS data to suggest ways to earn points based on the user's current location.

Description

Technical Field

[0004] ,

[0006] , , ,

[0005] , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there is a problem that it is difficult for a user to grasp point campaigns and special sale information in real time, and it is difficult to efficiently obtain points.

[0005] The system according to the embodiment aims to enable a user to grasp point campaigns and special sale information in real time and efficiently obtain points.

Means for Solving the Problems

[0006] The system according to this embodiment comprises a recognition unit, a notification unit, a voice recognition unit, a voice notification unit, and a suggestion unit. The recognition unit recognizes products and advertisements captured by the camera. The notification unit notifies the user of point campaigns based on the products and advertisements recognized by the recognition unit. The voice recognition unit recognizes in-store announcements and conversations. The voice notification unit notifies the user of special sale information and campaigns recognized by the voice recognition unit. The suggestion unit uses GPS data to suggest ways to earn points based on the user's current location. [Effects of the Invention]

[0007] The system according to this embodiment allows users to stay informed about point campaigns and special offers in real time and efficiently earn points. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The point acquisition suggestion system according to an embodiment of the present invention is a system in which an AI agent recognizes the surrounding environment through the camera and microphone of a smartphone and suggests point acquisition opportunities in real time based on visual and auditory information. This point acquisition suggestion system instantly recognizes products and advertisements captured by the camera and notifies the user of related point campaigns. It also detects special sale information and campaigns from in-store announcements and conversations and notifies the user of point acquisition opportunities by voice. Furthermore, it utilizes GPS data to suggest the optimal way to acquire points based on the user's current location. For example, the point acquisition suggestion system overlays point-acquiring locations and products onto the actual scenery seen through the camera. For example, point-acquiring products and locations are displayed in AR on the scenery seen by the user through the camera. The point acquisition suggestion system learns the user's past behavior patterns and preferences and suggests point acquisition actions optimized for the individual. For example, it suggests the optimal way to acquire points based on the user's history of previously used point campaigns. The point acquisition suggestion system calculates the points acquired in real time and reports them by voice or push notification. For example, it notifies the user of the number of points acquired in real time when the user acquires points. This system aims to maximize point acquisition by analyzing every aspect of a user's daily life in real time. Offline, it integrates cameras, microphones, and location sensors to analyze 360-degree information around the user in real time and suggest point acquisition opportunities. Online, it centrally manages point systems from different companies and industries and suggests the most valuable ways to earn points. As a result, the point acquisition suggestion system can recognize the user's surroundings and suggest point acquisition opportunities in real time.

[0029] The point acquisition suggestion system according to this embodiment comprises a recognition unit, a notification unit, a voice recognition unit, a voice notification unit, and a suggestion unit. The recognition unit recognizes products and advertisements captured by a camera. The recognition unit instantly recognizes products and advertisements captured by a camera, for example. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, the recognition unit recognizes logos and text of products and advertisements and notifies users of point campaigns based on that. The notification unit notifies users of point campaigns based on products and advertisements recognized by the recognition unit. The notification unit, for example, notifies users of point campaign information via push notifications. The notification unit uses AI to make notifications at the optimal timing based on the user's interests and behavior patterns. For example, if a user shows interest in a particular product, the notification unit notifies them of a point campaign related to that product. The voice recognition unit recognizes in-store announcements and conversations. The voice recognition unit detects special sale information and campaigns from in-store announcements and conversations, for example. The voice recognition unit uses AI to analyze voice data and extract specific keywords and phrases. For example, the voice recognition unit recognizes announcements of special offers and campaigns and notifies the user of point-earning opportunities based on that. The voice notification unit notifies the user of the special offers and campaigns recognized by the voice recognition unit. For example, the voice notification unit informs the user of point-earning opportunities by voice. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. For example, if the user prefers a particular voice notification, the voice notification unit prioritizes that notification. The suggestion unit uses GPS data to suggest point-earning methods based on the user's current location. For example, when the user approaches a specific store, the suggestion unit notifies the user of point campaign information available at that store. The suggestion unit uses AI to analyze the user's location information and suggest the optimal point-earning method. For example, if the user is in a specific area, the suggestion unit suggests point-earning methods available in that area. As a result, the point-earning suggestion system according to this embodiment can recognize the user's surrounding environment and suggest point-earning opportunities in real time.

[0030] The recognition unit recognizes products and advertisements captured by the camera. For example, the recognition unit instantly recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. Specifically, the recognition unit utilizes image recognition technology to extract features such as logos, text, colors, and shapes of products and advertisements with high accuracy. For example, it can read barcodes and 2D codes (e.g., QR codes) printed on products and obtain product information based on them. Furthermore, the recognition unit utilizes image classification algorithms using deep learning to identify the type and brand of products and advertisements. As a result, the recognition unit can quickly and accurately recognize objects pointed at by the user and extract relevant point campaign information. The recognition unit processes image data in real time and automatically generates recognition results without waiting for user input. This allows users to smoothly obtain opportunities to earn points. In addition, the recognition unit maintains recognition accuracy by linking with a cloud-based database and constantly updating the latest advertisement and product information. As a result, the recognition unit can always perform accurate recognition based on the latest information.

[0031] The notification unit notifies users of point campaigns based on products and advertisements recognized by the recognition unit. For example, the notification unit sends push notifications to users regarding point campaign information. The notification unit uses AI to send notifications at the optimal time based on the user's interests and behavior patterns. Specifically, the notification unit analyzes the user's past purchase and browsing history to identify products and campaigns that the user is likely to be interested in. For example, if a user has previously purchased products from a particular brand, the notification unit will prioritize notifying them of new point campaigns from that brand. The notification unit also learns the user's behavior patterns to identify the times and situations in which the user is most likely to receive notifications. This allows the notification unit to provide point campaign information at the time when the user is most likely to be interested. Furthermore, the notification unit maximizes the effectiveness of notifications by personalizing the content and using different messages and images for each user. For example, using messages that include the user's name or images of products that the user is likely to be interested in can improve the open rate and response rate of notifications. As a result, the notification unit can provide users with effective and attractive point campaign information and increase user engagement.

[0032] The speech recognition unit recognizes in-store announcements and conversations. For example, it can detect special sale information and campaigns from in-store announcements and conversations. The speech recognition unit uses AI to analyze audio data and extract specific keywords and phrases. Specifically, the speech recognition unit utilizes a deep learning-based speech recognition algorithm to analyze audio data from in-store announcements and conversations with high accuracy. For example, it can detect keywords such as "sale," "discount," and "points" included in announcements about special sales and campaigns in real time and notify users of opportunities to earn points based on that. In addition, the speech recognition unit uses noise reduction technology to remove in-store noise and background sounds, obtaining clear audio data. This allows the speech recognition unit to maintain high recognition accuracy even in noisy environments. Furthermore, the speech recognition unit can analyze multiple audio sources simultaneously and integrate information from different announcements and conversations. This allows the speech recognition unit to centrally manage announcements and conversations taking place in multiple locations within the store and provide users with comprehensive point earning information.

[0033] The voice notification unit notifies users of special offers and campaigns recognized by the voice recognition unit. For example, the voice notification unit informs users of point-earning opportunities via voice. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. Specifically, the voice notification unit analyzes the user's past voice notification history and feedback to identify the voice tone, speed, and language the user prefers. For example, if the user prefers voice notifications in a calm tone, it will notify them of point-earning opportunities in that tone. The voice notification unit also adjusts the content and timing of voice notifications according to the user's current situation and environment. For example, if the user is in a quiet place, it can lower the volume of the voice notification or switch to a vibration notification. Furthermore, the voice notification unit can combine multiple voice notifications to provide users with comprehensive information. For example, it can notify users of special offer information and campaign details in stages, ensuring that the user understands all the necessary information. In this way, the voice notification unit can provide users with effective and personalized voice notifications, ensuring they don't miss out on point-earning opportunities.

[0034] The proposal department utilizes GPS data to suggest point-earning methods based on the user's current location. For example, when a user approaches a specific store, the proposal department will notify them of point campaigns available at that store. The proposal department uses AI to analyze the user's location information and suggest the optimal point-earning method. Specifically, the proposal department analyzes the user's current location and travel history to predict stores and areas the user is likely to visit. For example, it identifies shopping malls and shopping streets that the user frequently visits and prioritizes notifying them of point campaigns available in those areas. Furthermore, the proposal department suggests point-earning methods in real time based on the user's current location. For example, when a user approaches a specific store, it immediately notifies them of point campaigns being held at that store. In addition, the proposal department makes personalized suggestions considering the user's past behavior patterns and interests. For example, if a user has previously purchased products from a specific brand, it will prioritize suggesting new point campaigns from that brand. This allows the proposal department to provide users with the optimal point-earning method and increase user engagement. Furthermore, the proposal department can collect user feedback and continuously improve the accuracy and effectiveness of its proposals. This allows the proposal department to consistently provide highly accurate proposals based on the latest information, thereby increasing user satisfaction.

[0035] The point acquisition suggestion system includes an AR display unit that overlays point-earning locations and products onto the actual scenery viewed through the camera. The AR display unit, for example, displays point-earning products and locations in AR on the scenery viewed through the camera. The AR display unit uses AI to overlay point-earning information onto the image captured by the camera. For example, the AR display unit overlays point campaign information onto products viewed by the user through the camera. The AR display unit displays optimal point-earning information based on the user's location information and past behavior patterns. For example, the AR display unit displays optimal point-earning information based on the user's history of participating in point campaigns in the past. This allows the user to visually confirm point-earning locations and products. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input the image captured by the camera into a generating AI and have the generating AI perform the overlay display of point-earning information.

[0036] The point acquisition suggestion system includes a learning unit that learns the user's past behavior patterns and preferences and proposes point acquisition actions optimized for the individual. The learning unit, for example, learns the user's past behavior patterns and preferences. The learning unit uses AI to analyze the user's behavior data and extract specific patterns and preferences. For example, the learning unit proposes the optimal way to acquire points based on the user's purchase and browsing history. The learning unit proposes the optimal point acquisition action based on the user's behavior patterns. For example, the learning unit proposes the optimal way to acquire points based on the user's history of participating in point campaigns in the past. This enables the system to propose point acquisition actions optimized for the user. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input the user's behavior data into a generating AI and have the generating AI execute a proposal for the optimal point acquisition action.

[0037] The point acquisition suggestion system includes a calculation unit that calculates the points earned in real time and reports them via voice or push notification. The calculation unit, for example, calculates the points earned in real time. The calculation unit uses AI to calculate points and notify the user. For example, when a user earns points, the calculation unit notifies the user of the number of points in real time. The calculation unit calculates points based on the user's behavior data and information on point campaigns. For example, when a user purchases a specific product, the calculation unit calculates and notifies the user of the points associated with that product. This allows the user to check their point acquisition status in real time. Some or all of the above processing in the calculation unit may be performed using AI or not using AI. For example, the calculation unit can input user behavior data into a generating AI and have the generating AI perform the point calculation.

[0038] The recognition unit instantly recognizes products and advertisements captured by the camera. For example, the recognition unit instantly recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, the recognition unit recognizes logos and text on products and advertisements and notifies users of point campaigns based on that. The recognition unit improves recognition accuracy based on user behavior data and past recognition history. For example, the recognition unit optimizes its recognition algorithm based on data of products and advertisements that the user has recognized in the past. This enables rapid notification of point campaigns by instantly recognizing products and advertisements captured by the camera. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input image data of products and advertisements captured by the camera into a generating AI and have the generating AI perform the recognition.

[0039] The speech recognition unit detects special sale information and campaigns from in-store announcements and conversations. For example, the speech recognition unit detects special sale information and campaigns from in-store announcements and conversations. The speech recognition unit uses AI to analyze the audio data and extract specific keywords and phrases. For example, the speech recognition unit recognizes announcements about special sales and campaigns and notifies users of opportunities to earn points based on that recognition. The speech recognition unit improves recognition accuracy based on user behavior data and past recognition history. For example, the speech recognition unit optimizes its recognition algorithm based on data of special sale information and campaigns that the user has recognized in the past. This allows for rapid notification to users by detecting special sale information and campaigns from in-store announcements and conversations. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input audio data of in-store announcements and conversations into a generating AI and have the generating AI perform the recognition.

[0040] The recognition unit applies different recognition algorithms depending on the brand and category of the product during recognition. For example, the recognition unit applies a more detailed recognition algorithm to luxury brand products. For food products, the recognition unit applies a recognition algorithm that takes freshness and expiration date into consideration. For electronic devices, the recognition unit applies an algorithm that recognizes the model number and specifications in detail. By applying a recognition algorithm according to the brand and category of the product, recognition accuracy is improved. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input brand and category data of the product into a generating AI and have the generating AI execute the application of the recognition algorithm.

[0041] The recognition unit determines the recognition priority based on the frequency of use and popularity of the products during recognition. For example, the recognition unit prioritizes recognizing products with high usage frequency. The recognition unit prioritizes recognizing products with high popularity. The recognition unit prioritizes recognizing products that are both frequently used and highly popular. In this way, by determining the recognition priority based on the frequency of use and popularity of the products, important products can be recognized preferentially. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input data on the frequency of use and popularity of products into a generating AI and have the generating AI perform the determination of the recognition priority.

[0042] The recognition unit optimizes its recognition algorithm based on the product's color and shape during recognition. For example, for products where color identification is important, the recognition unit enhances its color recognition algorithm. For products where shape identification is important, the recognition unit enhances its shape recognition algorithm. For products where both color and shape are important, the recognition unit optimizes both recognition algorithms. This improves recognition accuracy by optimizing the recognition algorithm based on the product's color and shape. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input color and shape data of the product into a generating AI and have the generating AI perform the optimization of the recognition algorithm.

[0043] The recognition unit acquires relevant product information in real time during recognition and reflects it in the recognition results. For example, the recognition unit acquires the latest price information of a product in real time and reflects it in the recognition results. The recognition unit acquires the inventory status of a product in real time and reflects it in the recognition results. The recognition unit acquires product reviews and ratings in real time and reflects them in the recognition results. In this way, by acquiring relevant product information in real time and reflecting it in the recognition results, the latest information can be provided. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input relevant product information into a generating AI and have the generating AI perform the reflection of it in the recognition results.

[0044] The notification unit determines the priority of notifications based on the importance of the point campaigns when sending notifications. For example, the notification unit prioritizes notifications for high-point campaigns. The notification unit prioritizes notifications for limited-time campaigns. The notification unit determines the importance and priority of notifications based on the user's interests. This allows important campaigns to be notified preferentially by determining the priority of notifications based on the importance of the point campaigns. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data on the importance of point campaigns into a generating AI and have the generating AI perform the determination of notification priorities.

[0045] The notification unit, when issuing a notification, selects the optimal notification method by referring to the user's past notification history. For example, the notification unit prioritizes notification methods that the user has preferred to receive in the past. The notification unit avoids notification methods that the user has ignored in the past. The notification unit analyzes the user's past notification history and selects the optimal notification method. This allows for optimal notifications for the user by selecting the optimal notification method by referring to the user's past notification history. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data from the user's past notification history into a generating AI and have the generating AI select the optimal notification method.

[0046] The notification unit selects the optimal notification method when a notification is sent, taking into account the user's device information. For example, if the user is using a smartphone, the notification unit prioritizes push notifications. If the user is using a tablet, the notification unit provides a notification method optimized for a larger screen. If the user is using a smartwatch, the notification unit provides a concise and highly visible notification method. By selecting the optimal notification method considering the user's device information, the notification unit can provide the most suitable notification for the user. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input the user's device information into a generating AI and have the generating AI select the optimal notification method.

[0047] The notification unit adjusts the notification frequency when sending notifications, taking into account the expiration date of the point campaign. For example, the notification unit prioritizes notifications for campaigns with approaching expiration dates. The notification unit reduces the notification frequency for campaigns with longer expiration dates. The notification unit sends notifications at the optimal time according to the user's schedule. In this way, by adjusting the notification frequency considering the expiration date of the point campaign, notifications can be sent at the appropriate time. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the adjustment of the notification frequency.

[0048] The speech recognition unit determines the recognition priority based on the importance of special offer information and campaigns during speech recognition. For example, the speech recognition unit prioritizes the recognition of campaign information with high points. The speech recognition unit prioritizes the recognition of limited-time special offer information. The speech recognition unit determines the importance and recognition priority based on the user's interests. In this way, important information can be recognized preferentially by determining the recognition priority based on the importance of special offer information and campaigns. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the importance of special offer information and campaigns into a generating AI and have the generating AI perform the determination of the recognition priority.

[0049] The speech recognition unit optimizes its recognition algorithm according to the noise level in the store during speech recognition. For example, if the store is quiet, the speech recognition unit uses a normal speech recognition algorithm. If the store is noisy, the speech recognition unit uses a speech recognition algorithm with enhanced noise cancellation. The speech recognition unit adjusts the speech recognition algorithm in real time according to the noise level in the store. This improves recognition accuracy by optimizing the recognition algorithm according to the noise level in the store. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the noise level in the store into a generating AI and have the generating AI perform the optimization of the recognition algorithm.

[0050] The speech recognition unit applies different recognition algorithms depending on the category of the sale information or campaign during speech recognition. For example, the speech recognition unit applies a speech recognition algorithm specialized for food to sale information in the food category. For campaign information in the electronics category, the speech recognition unit applies a speech recognition algorithm specialized for electronics. For sale information in the clothing category, the speech recognition unit applies a speech recognition algorithm specialized for clothing. By applying different recognition algorithms depending on the category of the sale information or campaign, recognition accuracy is improved. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the category of sale information or campaign into a generating AI and have the generating AI execute the application of the recognition algorithm.

[0051] The speech recognition unit analyzes the store's acoustic environment in real time during speech recognition to improve recognition accuracy. For example, the speech recognition unit analyzes the store's acoustic environment in real time and applies the optimal speech recognition algorithm. The speech recognition unit enhances the noise cancellation function according to the noise level in the store. The speech recognition unit adjusts the accuracy of speech recognition in real time based on the store's acoustic environment. This enables accurate speech recognition by analyzing the store's acoustic environment in real time and improving recognition accuracy. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the store's acoustic environment into a generating AI and have the generating AI perform the improvement of recognition accuracy.

[0052] The voice notification unit determines the priority of notifications based on the importance of special offers and campaigns when an audio notification is sent. For example, the voice notification unit prioritizes sending audio notifications about high-point campaigns. The voice notification unit also prioritizes sending audio notifications about limited-time special offers. The voice notification unit determines the priority of notifications based on the importance of the user's interests. This allows important information to be notified preferentially by determining the priority of notifications based on the importance of special offers and campaigns. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data on the importance of special offers and campaigns into a generating AI and have the generating AI perform the determination of notification priorities.

[0053] The voice notification unit, when issuing a voice notification, selects the optimal notification method by referring to the user's past notification history. For example, the voice notification unit prioritizes voice notification methods that the user has preferred to receive in the past. The voice notification unit avoids voice notification methods that the user has ignored in the past. The voice notification unit analyzes the user's past notification history and selects the optimal voice notification method. This makes it possible to provide the most suitable notification to the user by selecting the optimal notification method by referring to the user's past notification history. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data of the user's past notification history into a generating AI and have the generating AI perform the selection of the optimal notification method.

[0054] The voice notification unit selects the optimal notification method when an audio notification is sent, taking into account the user's device information. For example, if the user is using a smartphone, the voice notification unit provides an audio notification method optimized for smartphones. If the user is using a tablet, the voice notification unit provides an audio notification method optimized for tablets. If the user is using a smartwatch, the voice notification unit provides an audio notification method optimized for smartwatches. By selecting the optimal notification method while considering the user's device information, it becomes possible to provide notifications that are optimal for the user. Some or all of the above processing in the voice notification unit may be performed using AI, or it may be performed without using AI. For example, the voice notification unit can input the user's device information into a generating AI and have the generating AI perform the selection of the optimal notification method.

[0055] The voice notification unit adjusts the frequency of notifications when sending voice notifications, taking into account the expiration dates of special offers and campaigns. For example, the voice notification unit prioritizes sending voice notifications for campaigns with approaching expiration dates. The voice notification unit reduces the frequency of notifications for campaigns with longer expiration dates. The voice notification unit sends voice notifications at the optimal time according to the user's schedule. This allows notifications to be sent at the appropriate time by adjusting the notification frequency while considering the expiration dates of special offers and campaigns. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data on the expiration dates of special offers and campaigns into a generating AI and have the generating AI adjust the notification frequency.

[0056] The suggestion unit selects the optimal point-earning method based on the user's current location when making a suggestion. For example, when a user approaches a specific store, the suggestion unit notifies the user of point campaigns available at that store. If the user is in a specific area, the suggestion unit suggests point-earning methods available in that area. The suggestion unit suggests the optimal point-earning method in real time based on the user's current location. This enables efficient point earning by selecting the optimal point-earning method based on the user's current location. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input the user's current location data into a generating AI and have the generating AI select the optimal point-earning method.

[0057] The suggestion unit makes optimal suggestions by referring to the user's past behavior patterns. For example, the suggestion unit suggests the best way to earn points based on the user's history of participating in point campaigns. The suggestion unit analyzes the user's past behavior patterns and suggests the most efficient way to earn points. The suggestion unit makes optimal suggestions in real time based on the user's past behavior patterns. This makes it possible to make optimal suggestions for the user by referring to the user's past behavior patterns. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input data on the user's past behavior patterns into a generating AI and have the generating AI execute the optimal suggestion.

[0058] The suggestion unit selects the optimal suggestion method when making a suggestion, taking into account the user's device information. For example, if the user is using a smartphone, the suggestion unit provides a suggestion method optimized for smartphones. If the user is using a tablet, the suggestion unit provides a suggestion method optimized for tablets. If the user is using a smartwatch, the suggestion unit provides a suggestion method optimized for smartwatches. By selecting the optimal suggestion method while considering the user's device information, it becomes possible to make the best possible suggestions for the user. Some or all of the above processing in the suggestion unit may be performed using AI, or it may be performed without AI. For example, the suggestion unit can input the user's device information into a generating AI and have the generating AI select the optimal suggestion method.

[0059] The proposal unit adjusts the frequency of proposals, taking into account the expiration dates of point campaigns. For example, the proposal unit prioritizes proposing campaigns with approaching expiration dates. It reduces the frequency of proposals for campaigns with longer expiration dates. The proposal unit makes proposals at the optimal time according to the user's schedule. By adjusting the frequency of proposals, taking into account the expiration dates of point campaigns, proposals can be made at the appropriate time. Some or all of the above processing in the proposal unit may be performed using AI or not. For example, the proposal unit can input data on the expiration dates of point campaigns into a generation AI and have the generation AI adjust the frequency of proposals.

[0060] The AR display unit determines the display priority based on the importance of locations and products where points can be earned during AR display. For example, the AR display unit may prioritize displaying locations and products that offer high points in AR. The AR display unit may also prioritize displaying locations and products where points can be earned for a limited time in AR. The AR display unit determines the display priority based on the importance of the user's interests. This allows important information to be displayed preferentially by determining the display priority based on the importance of locations and products where points can be earned. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input data on the importance of locations and products where points can be earned into a generating AI and have the generating AI perform the determination of the display priority.

[0061] The AR display unit selects the optimal display method by referring to the user's past behavior patterns when displaying AR content. For example, the AR display unit selects the optimal AR display method based on the point acquisition methods the user has used in the past. The AR display unit analyzes the user's past behavior patterns and selects the most efficient AR display method. The AR display unit selects the optimal AR display method in real time based on the user's past behavior patterns. This makes it possible to display content optimally for the user by selecting the optimal display method by referring to the user's past behavior patterns. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input data on the user's past behavior patterns into a generating AI and have the generating AI perform the selection of the optimal display method.

[0062] The AR display unit selects the optimal display method when displaying AR content, taking into account the user's device information. For example, if the user is using a smartphone, the AR display unit provides an AR display method optimized for smartphones. If the user is using a tablet, the AR display unit provides an AR display method optimized for tablets. If the user is using a smartwatch, the AR display unit provides an AR display method optimized for smartwatches. By selecting the optimal display method considering the user's device information, the optimal display for the user becomes possible. Some or all of the above processing in the AR display unit may be performed using AI, or it may be performed without using AI. For example, the AR display unit can input the user's device information into a generating AI and have the generating AI select the optimal display method.

[0063] The AR display unit adjusts the display frequency when displaying AR content, taking into account the expiration dates of point campaigns. For example, the AR display unit prioritizes displaying campaigns with approaching expiration dates. The AR display unit reduces the display frequency of campaigns with longer expiration dates. The AR display unit displays AR content at the optimal timing according to the user's schedule. By adjusting the display frequency while considering the expiration dates of point campaigns, the display can be performed at the appropriate time. Some or all of the above processing in the AR display unit may be performed using AI, or not. For example, the AR display unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the adjustment of the display frequency.

[0064] The learning unit optimizes the learning algorithm by referring to past learning data during the learning process. For example, the learning unit analyzes past learning data and selects the optimal learning algorithm. The learning unit extracts effective learning patterns from past learning data. The learning unit optimizes the learning algorithm in real time based on past learning data. This improves the accuracy of learning by optimizing the learning algorithm by referring to past learning data. Some or all of the above processes in the learning unit may be performed using AI or not. For example, the learning unit can input past learning data into a generating AI and have the generating AI perform the optimization of the learning algorithm.

[0065] The learning unit determines learning priorities based on the user's behavior patterns during the learning process. For example, the learning unit prioritizes learning point-earning methods that the user frequently uses. The learning unit analyzes the user's behavior patterns and determines the most effective learning order. The learning unit adjusts learning priorities in real time based on the user's past behavior patterns. This enables efficient learning by determining learning priorities based on the user's behavior patterns. Some or all of the above processes in the learning unit may be performed using AI or not. For example, the learning unit can input user behavior pattern data into a generating AI and have the generating AI perform the determination of learning priorities.

[0066] The learning unit selects the optimal learning method during learning, taking into account the user's device information. For example, if the user is using a smartphone, the learning unit provides a learning method optimized for smartphones. If the user is using a tablet, the learning unit provides a learning method optimized for tablets. If the user is using a smartwatch, the learning unit provides a learning method optimized for smartwatches. By selecting the optimal learning method while considering the user's device information, the learning unit enables optimal learning for the user. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input the user's device information into a generating AI and have the generating AI select the optimal learning method.

[0067] The learning unit weights the training data during training, taking into account the expiration dates of point campaigns. For example, the learning unit gives higher weight to training data for campaigns with approaching expiration dates, and lower weight to training data for campaigns with longer expiration dates. The learning unit adjusts the weighting of the training data in real time according to the user's schedule. This allows training to be performed at the appropriate time by weighting the training data while considering the expiration dates of point campaigns. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the weighting of the training data.

[0068] The calculation unit determines the calculation priority based on the importance of the points earned when calculating points. For example, the calculation unit may prioritize the acquisition of high points. The calculation unit may prioritize the acquisition of points that are available for a limited time. The calculation unit determines the importance and calculation priority based on the user's interests. In this way, important points can be calculated preferentially by determining the calculation priority based on the importance of the points earned. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input data on the importance of the points earned into a generating AI and have the generating AI perform the determination of the calculation priority.

[0069] The calculation unit selects the optimal calculation method by referring to the user's past point history when calculating points. For example, the calculation unit selects the optimal calculation method based on the user's past point history. The calculation unit analyzes the user's past point history and selects the most efficient calculation method. The calculation unit optimizes the calculation method in real time based on the user's past point history. This enables efficient point calculation by selecting the optimal calculation method by referring to the user's past point history. Some or all of the above processes in the calculation unit may be performed using AI or not. For example, the calculation unit can input data of the user's past point history into a generating AI and have the generating AI perform the selection of the optimal calculation method.

[0070] The calculation unit selects the optimal calculation method when calculating points, taking into account the user's device information. For example, if the user is using a smartphone, the calculation unit provides a calculation method optimized for smartphones. If the user is using a tablet, the calculation unit provides a calculation method optimized for tablets. If the user is using a smartwatch, the calculation unit provides a calculation method optimized for smartwatches. By selecting the optimal calculation method while considering the user's device information, the calculation unit enables the user to perform calculations that are optimal for them. Some or all of the above processing in the calculation unit may be performed using AI, or it may be performed without AI. For example, the calculation unit can input the user's device information into a generating AI and have the generating AI perform the selection of the optimal calculation method.

[0071] The calculation unit adjusts the frequency of point calculations, taking into account the expiration dates of point campaigns. For example, the calculation unit prioritizes calculating points for campaigns with approaching expiration dates. It reduces the frequency of point calculations for campaigns with longer expiration dates. The calculation unit performs point calculations at the optimal time according to the user's schedule. By adjusting the frequency of calculations, taking into account the expiration dates of point campaigns, calculations can be performed at the appropriate time. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI adjust the calculation frequency.

[0072] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0073] The proposal department can acquire users' health data and suggest ways to earn points based on their health status. For example, it can acquire users' step count data and propose a campaign where users can earn points when they reach a certain number of steps. It can also acquire users' heart rate data and propose ways to earn points when they engage in moderate exercise. Furthermore, it can acquire users' sleep data and propose a campaign where users can earn points when they get enough sleep. In this way, by suggesting ways to earn points that are tailored to the user's health status, it is possible to promote health.

[0074] The proposal department can suggest ways for users to earn points based on their hobbies and interests. For example, if a user is interested in music, it can suggest ways to earn points related to music-related events and products. If a user is interested in sports, it can suggest ways to earn points related to sports events and sporting goods. Furthermore, if a user is interested in cooking, it can suggest ways to earn points related to cooking classes and ingredients. In this way, by suggesting ways to earn points that match the user's hobbies and interests, it is possible to attract the user's interest.

[0075] The suggestion function can propose ways for users to earn points based on their past purchase history. For example, it can suggest point campaigns related to products the user has purchased in the past. It can also suggest ways to earn points related to stores or brands that the user frequently uses. Furthermore, if a user tends to purchase products in a particular category, it can suggest ways to earn points related to that category. In this way, by suggesting ways to earn points based on the user's purchase history, it is possible to increase the user's motivation to purchase.

[0076] The proposal team can suggest ways for users to earn points based on their social media activity. For example, they can propose campaigns where users earn points when they share posts about specific brands or products. They can also suggest ways for users to earn points when they post using specific hashtags. Furthermore, they can suggest ways for users to earn points when they participate in specific events or campaigns. By suggesting point-earning methods based on users' social media activity, this can increase user engagement.

[0077] The proposal team can suggest ways for users to earn points based on their life events. For example, they can propose a special points campaign on a user's birthday. They can also suggest ways to earn points related to special days such as a user's wedding anniversary or a child's birthday. Furthermore, they can suggest ways to earn points related to life events such as moving or starting a new job. By suggesting ways to earn points that are tailored to a user's life events, they can make special days even more special for the user.

[0078] The proposal department can suggest ways for users to earn points based on their travel plans. For example, it can suggest point campaigns that users can use at their travel destination. It can also suggest ways to earn points related to tourist attractions and restaurants that users will visit during their trip. Furthermore, it can suggest ways to earn points related to items that users need to prepare before their trip. In this way, by suggesting ways to earn points that match the user's travel plans, the enjoyment of their trip can be further enhanced.

[0079] The following briefly describes the processing flow for example form 1.

[0080] Step 1: The recognition unit recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, it recognizes logos and text on products and advertisements and notifies users of point campaigns based on that recognition. Step 2: The notification unit notifies users of point campaigns based on the products and advertisements recognized by the recognition unit. The notification unit uses AI to send notifications at the optimal time based on the user's interests and behavioral patterns. For example, if a user shows interest in a particular product, it will notify them of point campaigns related to that product. Step 3: The voice recognition unit recognizes in-store announcements and conversations. The voice recognition unit uses AI to analyze the voice data and extract specific keywords and phrases. For example, it can detect special sale information and campaigns from in-store announcements and conversations. Step 4: The voice notification unit notifies the user of special offers and campaigns recognized by the voice recognition unit. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. For example, if the user prefers a particular voice notification, that notification will be prioritized. Step 5: The proposal team uses GPS data to suggest point-earning methods based on the user's current location. The proposal team uses AI to analyze the user's location information and propose the optimal point-earning method. For example, if the user is in a specific area, it will suggest point-earning methods available in that area.

[0081] (Example of form 2) The point acquisition suggestion system according to an embodiment of the present invention is a system in which an AI agent recognizes the surrounding environment through the camera and microphone of a smartphone and suggests point acquisition opportunities in real time based on visual and auditory information. This point acquisition suggestion system instantly recognizes products and advertisements captured by the camera and notifies the user of related point campaigns. It also detects special sale information and campaigns from in-store announcements and conversations and notifies the user of point acquisition opportunities by voice. Furthermore, it utilizes GPS data to suggest the optimal way to acquire points based on the user's current location. For example, the point acquisition suggestion system overlays point-acquiring locations and products onto the actual scenery seen through the camera. For example, point-acquiring products and locations are displayed in AR on the scenery seen by the user through the camera. The point acquisition suggestion system learns the user's past behavior patterns and preferences and suggests point acquisition actions optimized for the individual. For example, it suggests the optimal way to acquire points based on the user's history of previously used point campaigns. The point acquisition suggestion system calculates the points acquired in real time and reports them by voice or push notification. For example, it notifies the user of the number of points acquired in real time when the user acquires points. This system aims to maximize point acquisition by analyzing every aspect of a user's daily life in real time. Offline, it integrates cameras, microphones, and location sensors to analyze 360-degree information around the user in real time and suggest point acquisition opportunities. Online, it centrally manages point systems from different companies and industries and suggests the most valuable ways to earn points. As a result, the point acquisition suggestion system can recognize the user's surroundings and suggest point acquisition opportunities in real time.

[0082] The point acquisition suggestion system according to this embodiment comprises a recognition unit, a notification unit, a voice recognition unit, a voice notification unit, and a suggestion unit. The recognition unit recognizes products and advertisements captured by a camera. The recognition unit instantly recognizes products and advertisements captured by a camera, for example. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, the recognition unit recognizes logos and text of products and advertisements and notifies users of point campaigns based on that. The notification unit notifies users of point campaigns based on products and advertisements recognized by the recognition unit. The notification unit, for example, notifies users of point campaign information via push notifications. The notification unit uses AI to make notifications at the optimal timing based on the user's interests and behavior patterns. For example, if a user shows interest in a particular product, the notification unit notifies them of a point campaign related to that product. The voice recognition unit recognizes in-store announcements and conversations. The voice recognition unit detects special sale information and campaigns from in-store announcements and conversations, for example. The voice recognition unit uses AI to analyze voice data and extract specific keywords and phrases. For example, the voice recognition unit recognizes announcements of special offers and campaigns and notifies the user of point-earning opportunities based on that. The voice notification unit notifies the user of the special offers and campaigns recognized by the voice recognition unit. For example, the voice notification unit informs the user of point-earning opportunities by voice. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. For example, if the user prefers a particular voice notification, the voice notification unit prioritizes that notification. The suggestion unit uses GPS data to suggest point-earning methods based on the user's current location. For example, when the user approaches a specific store, the suggestion unit notifies the user of point campaign information available at that store. The suggestion unit uses AI to analyze the user's location information and suggest the optimal point-earning method. For example, if the user is in a specific area, the suggestion unit suggests point-earning methods available in that area. As a result, the point-earning suggestion system according to this embodiment can recognize the user's surrounding environment and suggest point-earning opportunities in real time.

[0083] The recognition unit recognizes products and advertisements captured by the camera. For example, the recognition unit instantly recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. Specifically, the recognition unit utilizes image recognition technology to extract features such as logos, text, colors, and shapes of products and advertisements with high accuracy. For example, it can read barcodes and 2D codes (e.g., QR codes) printed on products and obtain product information based on them. Furthermore, the recognition unit utilizes image classification algorithms using deep learning to identify the type and brand of products and advertisements. As a result, the recognition unit can quickly and accurately recognize objects pointed at by the user and extract relevant point campaign information. The recognition unit processes image data in real time and automatically generates recognition results without waiting for user input. This allows users to smoothly obtain opportunities to earn points. In addition, the recognition unit maintains recognition accuracy by linking with a cloud-based database and constantly updating the latest advertisement and product information. As a result, the recognition unit can always perform accurate recognition based on the latest information.

[0084] The notification unit notifies users of point campaigns based on products and advertisements recognized by the recognition unit. For example, the notification unit sends push notifications to users regarding point campaign information. The notification unit uses AI to send notifications at the optimal time based on the user's interests and behavior patterns. Specifically, the notification unit analyzes the user's past purchase and browsing history to identify products and campaigns that the user is likely to be interested in. For example, if a user has previously purchased products from a particular brand, the notification unit will prioritize notifying them of new point campaigns from that brand. The notification unit also learns the user's behavior patterns to identify the times and situations in which the user is most likely to receive notifications. This allows the notification unit to provide point campaign information at the time when the user is most likely to be interested. Furthermore, the notification unit maximizes the effectiveness of notifications by personalizing the content and using different messages and images for each user. For example, using messages that include the user's name or images of products that the user is likely to be interested in can improve the open rate and response rate of notifications. As a result, the notification unit can provide users with effective and attractive point campaign information and increase user engagement.

[0085] The speech recognition unit recognizes in-store announcements and conversations. For example, it can detect special sale information and campaigns from in-store announcements and conversations. The speech recognition unit uses AI to analyze audio data and extract specific keywords and phrases. Specifically, the speech recognition unit utilizes a deep learning-based speech recognition algorithm to analyze audio data from in-store announcements and conversations with high accuracy. For example, it can detect keywords such as "sale," "discount," and "points" included in announcements about special sales and campaigns in real time and notify users of opportunities to earn points based on that. In addition, the speech recognition unit uses noise reduction technology to remove in-store noise and background sounds, obtaining clear audio data. This allows the speech recognition unit to maintain high recognition accuracy even in noisy environments. Furthermore, the speech recognition unit can analyze multiple audio sources simultaneously and integrate information from different announcements and conversations. This allows the speech recognition unit to centrally manage announcements and conversations taking place in multiple locations within the store and provide users with comprehensive point earning information.

[0086] The voice notification unit notifies users of special offers and campaigns recognized by the voice recognition unit. For example, the voice notification unit informs users of point-earning opportunities via voice. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. Specifically, the voice notification unit analyzes the user's past voice notification history and feedback to identify the voice tone, speed, and language the user prefers. For example, if the user prefers voice notifications in a calm tone, it will notify them of point-earning opportunities in that tone. The voice notification unit also adjusts the content and timing of voice notifications according to the user's current situation and environment. For example, if the user is in a quiet place, it can lower the volume of the voice notification or switch to a vibration notification. Furthermore, the voice notification unit can combine multiple voice notifications to provide users with comprehensive information. For example, it can notify users of special offer information and campaign details in stages, ensuring that the user understands all the necessary information. In this way, the voice notification unit can provide users with effective and personalized voice notifications, ensuring they don't miss out on point-earning opportunities.

[0087] The proposal department utilizes GPS data to suggest point-earning methods based on the user's current location. For example, when a user approaches a specific store, the proposal department will notify them of point campaigns available at that store. The proposal department uses AI to analyze the user's location information and suggest the optimal point-earning method. Specifically, the proposal department analyzes the user's current location and travel history to predict stores and areas the user is likely to visit. For example, it identifies shopping malls and shopping streets that the user frequently visits and prioritizes notifying them of point campaigns available in those areas. Furthermore, the proposal department suggests point-earning methods in real time based on the user's current location. For example, when a user approaches a specific store, it immediately notifies them of point campaigns being held at that store. In addition, the proposal department makes personalized suggestions considering the user's past behavior patterns and interests. For example, if a user has previously purchased products from a specific brand, it will prioritize suggesting new point campaigns from that brand. This allows the proposal department to provide users with the optimal point-earning method and increase user engagement. Furthermore, the proposal department can collect user feedback and continuously improve the accuracy and effectiveness of its proposals. This allows the proposal department to consistently provide highly accurate proposals based on the latest information, thereby increasing user satisfaction.

[0088] The point acquisition suggestion system includes an AR display unit that overlays point-earning locations and products onto the actual scenery viewed through the camera. The AR display unit, for example, displays point-earning products and locations in AR on the scenery viewed through the camera. The AR display unit uses AI to overlay point-earning information onto the image captured by the camera. For example, the AR display unit overlays point campaign information onto products viewed by the user through the camera. The AR display unit displays optimal point-earning information based on the user's location information and past behavior patterns. For example, the AR display unit displays optimal point-earning information based on the user's history of participating in point campaigns in the past. This allows the user to visually confirm point-earning locations and products. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input the image captured by the camera into a generating AI and have the generating AI perform the overlay display of point-earning information.

[0089] The point acquisition suggestion system includes a learning unit that learns the user's past behavior patterns and preferences and proposes point acquisition actions optimized for the individual. The learning unit, for example, learns the user's past behavior patterns and preferences. The learning unit uses AI to analyze the user's behavior data and extract specific patterns and preferences. For example, the learning unit proposes the optimal way to acquire points based on the user's purchase and browsing history. The learning unit proposes the optimal point acquisition action based on the user's behavior patterns. For example, the learning unit proposes the optimal way to acquire points based on the user's history of participating in point campaigns in the past. This enables the system to propose point acquisition actions optimized for the user. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input the user's behavior data into a generating AI and have the generating AI execute a proposal for the optimal point acquisition action.

[0090] The point acquisition suggestion system includes a calculation unit that calculates the points earned in real time and reports them via voice or push notification. The calculation unit, for example, calculates the points earned in real time. The calculation unit uses AI to calculate points and notify the user. For example, when a user earns points, the calculation unit notifies the user of the number of points in real time. The calculation unit calculates points based on the user's behavior data and information on point campaigns. For example, when a user purchases a specific product, the calculation unit calculates and notifies the user of the points associated with that product. This allows the user to check their point acquisition status in real time. Some or all of the above processing in the calculation unit may be performed using AI or not using AI. For example, the calculation unit can input user behavior data into a generating AI and have the generating AI perform the point calculation.

[0091] The recognition unit instantly recognizes products and advertisements captured by the camera. For example, the recognition unit instantly recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, the recognition unit recognizes logos and text on products and advertisements and notifies users of point campaigns based on that. The recognition unit improves recognition accuracy based on user behavior data and past recognition history. For example, the recognition unit optimizes its recognition algorithm based on data of products and advertisements that the user has recognized in the past. This enables rapid notification of point campaigns by instantly recognizing products and advertisements captured by the camera. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input image data of products and advertisements captured by the camera into a generating AI and have the generating AI perform the recognition.

[0092] The speech recognition unit detects special sale information and campaigns from in-store announcements and conversations. For example, the speech recognition unit detects special sale information and campaigns from in-store announcements and conversations. The speech recognition unit uses AI to analyze the audio data and extract specific keywords and phrases. For example, the speech recognition unit recognizes announcements about special sales and campaigns and notifies users of opportunities to earn points based on that recognition. The speech recognition unit improves recognition accuracy based on user behavior data and past recognition history. For example, the speech recognition unit optimizes its recognition algorithm based on data of special sale information and campaigns that the user has recognized in the past. This allows for rapid notification to users by detecting special sale information and campaigns from in-store announcements and conversations. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input audio data of in-store announcements and conversations into a generating AI and have the generating AI perform the recognition.

[0093] The recognition unit estimates the user's emotions and adjusts the recognition accuracy based on the estimated emotions. For example, if the user is stressed, the recognition unit increases recognition accuracy to reduce misrecognition. If the user is relaxed, the recognition unit maintains the recognition accuracy at a normal setting. If the user is in a hurry, the recognition unit prioritizes recognition speed and tolerates some misrecognition. This reduces misrecognition by adjusting the recognition accuracy according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input user emotion data into a generative AI and have the generative AI perform the adjustment of recognition accuracy.

[0094] The recognition unit applies different recognition algorithms depending on the brand and category of the product during recognition. For example, the recognition unit applies a more detailed recognition algorithm to luxury brand products. For food products, the recognition unit applies a recognition algorithm that takes freshness and expiration date into consideration. For electronic devices, the recognition unit applies an algorithm that recognizes the model number and specifications in detail. By applying a recognition algorithm according to the brand and category of the product, recognition accuracy is improved. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input brand and category data of the product into a generating AI and have the generating AI execute the application of the recognition algorithm.

[0095] The recognition unit determines the recognition priority based on the frequency of use and popularity of the products during recognition. For example, the recognition unit prioritizes recognizing products with high usage frequency. The recognition unit prioritizes recognizing products with high popularity. The recognition unit prioritizes recognizing products that are both frequently used and highly popular. In this way, by determining the recognition priority based on the frequency of use and popularity of the products, important products can be recognized preferentially. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input data on the frequency of use and popularity of products into a generating AI and have the generating AI perform the determination of the recognition priority.

[0096] The recognition unit estimates the user's emotions and adjusts the display method of the recognition results based on the estimated emotions. For example, if the user is stressed, the recognition unit provides a simple and highly visible display method. If the user is relaxed, the recognition unit provides a display method that includes detailed information. If the user is in a hurry, the recognition unit provides a display method that gets straight to the point. By adjusting the display method of the recognition results according to the user's emotions, it becomes possible to provide a display that is easy for the user to understand. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to these examples. Some or all of the above processing in the recognition unit may be performed using AI or not using AI. For example, the recognition unit can input user emotion data into the generative AI and have the generative AI perform the adjustment of the display method of the recognition results.

[0097] The recognition unit optimizes its recognition algorithm based on the product's color and shape during recognition. For example, for products where color identification is important, the recognition unit enhances its color recognition algorithm. For products where shape identification is important, the recognition unit enhances its shape recognition algorithm. For products where both color and shape are important, the recognition unit optimizes both recognition algorithms. This improves recognition accuracy by optimizing the recognition algorithm based on the product's color and shape. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input color and shape data of the product into a generating AI and have the generating AI perform the optimization of the recognition algorithm.

[0098] The recognition unit acquires relevant product information in real time during recognition and reflects it in the recognition results. For example, the recognition unit acquires the latest price information of a product in real time and reflects it in the recognition results. The recognition unit acquires the inventory status of a product in real time and reflects it in the recognition results. The recognition unit acquires product reviews and ratings in real time and reflects them in the recognition results. In this way, by acquiring relevant product information in real time and reflecting it in the recognition results, the latest information can be provided. Some or all of the above processing in the recognition unit may be performed using AI or not. For example, the recognition unit can input relevant product information into a generating AI and have the generating AI perform the reflection of it in the recognition results.

[0099] The notification unit estimates the user's emotions and adjusts the timing of notifications based on the estimated emotions. For example, if the user is stressed, the notification unit reduces the frequency of notifications. If the user is relaxed, the notification unit maintains the normal notification frequency. If the user is in a hurry, the notification unit prioritizes only important notifications. This allows notifications to be delivered at the appropriate time by adjusting the timing according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input user emotion data into a generative AI and have the generative AI adjust the timing of notifications.

[0100] The notification unit determines the priority of notifications based on the importance of the point campaigns when sending notifications. For example, the notification unit prioritizes notifications for high-point campaigns. The notification unit prioritizes notifications for limited-time campaigns. The notification unit determines the importance and priority of notifications based on the user's interests. This allows important campaigns to be notified preferentially by determining the priority of notifications based on the importance of the point campaigns. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data on the importance of point campaigns into a generating AI and have the generating AI perform the determination of notification priorities.

[0101] The notification unit, when issuing a notification, selects the optimal notification method by referring to the user's past notification history. For example, the notification unit prioritizes notification methods that the user has preferred to receive in the past. The notification unit avoids notification methods that the user has ignored in the past. The notification unit analyzes the user's past notification history and selects the optimal notification method. This allows for optimal notifications for the user by selecting the optimal notification method by referring to the user's past notification history. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data from the user's past notification history into a generating AI and have the generating AI select the optimal notification method.

[0102] The notification unit estimates the user's emotions and customizes the notification content based on the estimated emotions. For example, if the user is stressed, the notification unit will create a concise and to-the-point notification. If the user is relaxed, the notification unit will create a notification that includes detailed information. If the user is in a hurry, the notification unit will include only the most important information. In this way, by customizing the notification content according to the user's emotions, the system can provide the user with the most appropriate notification. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input user emotion data into a generative AI and have the generative AI customize the notification content.

[0103] The notification unit selects the optimal notification method when a notification is sent, taking into account the user's device information. For example, if the user is using a smartphone, the notification unit prioritizes push notifications. If the user is using a tablet, the notification unit provides a notification method optimized for a larger screen. If the user is using a smartwatch, the notification unit provides a concise and highly visible notification method. By selecting the optimal notification method considering the user's device information, the notification unit can provide the most suitable notification for the user. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input the user's device information into a generating AI and have the generating AI select the optimal notification method.

[0104] The notification unit adjusts the notification frequency when sending notifications, taking into account the expiration date of the point campaign. For example, the notification unit prioritizes notifications for campaigns with approaching expiration dates. The notification unit reduces the notification frequency for campaigns with longer expiration dates. The notification unit sends notifications at the optimal time according to the user's schedule. In this way, by adjusting the notification frequency considering the expiration date of the point campaign, notifications can be sent at the appropriate time. Some or all of the above processing in the notification unit may be performed using AI or not. For example, the notification unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the adjustment of the notification frequency.

[0105] The speech recognition unit estimates the user's emotions and adjusts the accuracy of speech recognition based on the estimated emotions. For example, if the user is stressed, the speech recognition unit increases the accuracy of speech recognition to reduce misrecognition. If the user is relaxed, the speech recognition unit maintains the accuracy at a normal setting. If the user is in a hurry, the speech recognition unit prioritizes speech recognition speed and tolerates some misrecognition. This reduces misrecognition by adjusting the accuracy of speech recognition according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input user emotion data into the generative AI and have the generative AI perform the adjustment of speech recognition accuracy.

[0106] The speech recognition unit determines the recognition priority based on the importance of special offer information and campaigns during speech recognition. For example, the speech recognition unit prioritizes the recognition of campaign information with high points. The speech recognition unit prioritizes the recognition of limited-time special offer information. The speech recognition unit determines the importance and recognition priority based on the user's interests. In this way, important information can be recognized preferentially by determining the recognition priority based on the importance of special offer information and campaigns. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the importance of special offer information and campaigns into a generating AI and have the generating AI perform the determination of the recognition priority.

[0107] The speech recognition unit optimizes its recognition algorithm according to the noise level in the store during speech recognition. For example, if the store is quiet, the speech recognition unit uses a normal speech recognition algorithm. If the store is noisy, the speech recognition unit uses a speech recognition algorithm with enhanced noise cancellation. The speech recognition unit adjusts the speech recognition algorithm in real time according to the noise level in the store. This improves recognition accuracy by optimizing the recognition algorithm according to the noise level in the store. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the noise level in the store into a generating AI and have the generating AI perform the optimization of the recognition algorithm.

[0108] The speech recognition unit estimates the user's emotions and adjusts the display method of the speech recognition results based on the estimated emotions. For example, if the user is stressed, the speech recognition unit provides a simple and highly visible display method. If the user is relaxed, the speech recognition unit provides a display method that includes detailed information. If the user is in a hurry, the speech recognition unit provides a display method that gets straight to the point. By adjusting the display method of the speech recognition results according to the user's emotions, it becomes possible to provide a display that is easy for the user to understand. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to these examples. Some or all of the above processing in the speech recognition unit may be performed using AI or not using AI. For example, the speech recognition unit can input user emotion data into the generative AI and have the generative AI perform the adjustment of the display method of the speech recognition results.

[0109] The speech recognition unit applies different recognition algorithms depending on the category of the sale information or campaign during speech recognition. For example, the speech recognition unit applies a speech recognition algorithm specialized for food to sale information in the food category. For campaign information in the electronics category, the speech recognition unit applies a speech recognition algorithm specialized for electronics. For sale information in the clothing category, the speech recognition unit applies a speech recognition algorithm specialized for clothing. By applying different recognition algorithms depending on the category of the sale information or campaign, recognition accuracy is improved. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the category of sale information or campaign into a generating AI and have the generating AI execute the application of the recognition algorithm.

[0110] The speech recognition unit analyzes the store's acoustic environment in real time during speech recognition to improve recognition accuracy. For example, the speech recognition unit analyzes the store's acoustic environment in real time and applies the optimal speech recognition algorithm. The speech recognition unit enhances the noise cancellation function according to the noise level in the store. The speech recognition unit adjusts the accuracy of speech recognition in real time based on the store's acoustic environment. This enables accurate speech recognition by analyzing the store's acoustic environment in real time and improving recognition accuracy. Some or all of the above processing in the speech recognition unit may be performed using AI or not. For example, the speech recognition unit can input data on the store's acoustic environment into a generating AI and have the generating AI perform the improvement of recognition accuracy.

[0111] The voice notification unit estimates the user's emotions and adjusts the timing of voice notifications based on the estimated emotions. For example, if the user is stressed, the voice notification unit reduces the frequency of voice notifications. If the user is relaxed, the voice notification unit maintains the normal frequency of voice notifications. If the user is in a hurry, the voice notification unit prioritizes only important voice notifications. This allows notifications to be delivered at the appropriate time by adjusting the timing of voice notifications according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input user emotion data into a generative AI and have the generative AI adjust the timing of voice notifications.

[0112] The voice notification unit determines the priority of notifications based on the importance of special offers and campaigns when an audio notification is sent. For example, the voice notification unit prioritizes sending audio notifications about high-point campaigns. The voice notification unit also prioritizes sending audio notifications about limited-time special offers. The voice notification unit determines the priority of notifications based on the importance of the user's interests. This allows important information to be notified preferentially by determining the priority of notifications based on the importance of special offers and campaigns. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data on the importance of special offers and campaigns into a generating AI and have the generating AI perform the determination of notification priorities.

[0113] The voice notification unit, when issuing a voice notification, selects the optimal notification method by referring to the user's past notification history. For example, the voice notification unit prioritizes voice notification methods that the user has preferred to receive in the past. The voice notification unit avoids voice notification methods that the user has ignored in the past. The voice notification unit analyzes the user's past notification history and selects the optimal voice notification method. This makes it possible to provide the most suitable notification to the user by selecting the optimal notification method by referring to the user's past notification history. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data of the user's past notification history into a generating AI and have the generating AI perform the selection of the optimal notification method.

[0114] The voice notification unit estimates the user's emotions and customizes the content of the voice notification based on the estimated emotions. For example, if the user is stressed, the voice notification unit will produce a concise and to-the-point voice notification. If the user is relaxed, the voice notification unit will produce a voice notification that includes detailed information. If the user is in a hurry, the voice notification unit will include only the most important information in the voice notification. In this way, by customizing the content of the voice notification according to the user's emotions, the system can provide the user with the most appropriate notification. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input user emotion data into a generative AI and have the generative AI customize the content of the voice notification.

[0115] The voice notification unit selects the optimal notification method when an audio notification is sent, taking into account the user's device information. For example, if the user is using a smartphone, the voice notification unit provides an audio notification method optimized for smartphones. If the user is using a tablet, the voice notification unit provides an audio notification method optimized for tablets. If the user is using a smartwatch, the voice notification unit provides an audio notification method optimized for smartwatches. By selecting the optimal notification method while considering the user's device information, it becomes possible to provide notifications that are optimal for the user. Some or all of the above processing in the voice notification unit may be performed using AI, or it may be performed without using AI. For example, the voice notification unit can input the user's device information into a generating AI and have the generating AI perform the selection of the optimal notification method.

[0116] The voice notification unit adjusts the frequency of notifications when sending voice notifications, taking into account the expiration dates of special offers and campaigns. For example, the voice notification unit prioritizes sending voice notifications for campaigns with approaching expiration dates. The voice notification unit reduces the frequency of notifications for campaigns with longer expiration dates. The voice notification unit sends voice notifications at the optimal time according to the user's schedule. This allows notifications to be sent at the appropriate time by adjusting the notification frequency while considering the expiration dates of special offers and campaigns. Some or all of the above processing in the voice notification unit may be performed using AI or not. For example, the voice notification unit can input data on the expiration dates of special offers and campaigns into a generating AI and have the generating AI adjust the notification frequency.

[0117] The suggestion unit estimates the user's emotions and adjusts the content of the suggestions based on the estimated emotions. For example, if the user is stressed, the suggestion unit will make the suggestions concise and to the point. If the user is relaxed, the suggestion unit will make the suggestions include detailed information. If the user is in a hurry, the suggestion unit will include only the most important information. In this way, by adjusting the content of the suggestions according to the user's emotions, the system can provide the most suitable suggestions for the user. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input user emotion data into a generative AI and have the generative AI adjust the content of the suggestions.

[0118] The suggestion unit selects the optimal point-earning method based on the user's current location when making a suggestion. For example, when a user approaches a specific store, the suggestion unit notifies the user of point campaigns available at that store. If the user is in a specific area, the suggestion unit suggests point-earning methods available in that area. The suggestion unit suggests the optimal point-earning method in real time based on the user's current location. This enables efficient point earning by selecting the optimal point-earning method based on the user's current location. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input the user's current location data into a generating AI and have the generating AI select the optimal point-earning method.

[0119] The suggestion unit makes optimal suggestions by referring to the user's past behavior patterns. For example, the suggestion unit suggests the best way to earn points based on the user's history of participating in point campaigns. The suggestion unit analyzes the user's past behavior patterns and suggests the most efficient way to earn points. The suggestion unit makes optimal suggestions in real time based on the user's past behavior patterns. This makes it possible to make optimal suggestions for the user by referring to the user's past behavior patterns. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input data on the user's past behavior patterns into a generating AI and have the generating AI execute the optimal suggestion.

[0120] The suggestion unit estimates the user's emotions and adjusts the timing of suggestions based on the estimated emotions. For example, if the user is stressed, the suggestion unit reduces the frequency of suggestions. If the user is relaxed, the suggestion unit maintains the normal frequency of suggestions. If the user is in a hurry, the suggestion unit prioritizes only important suggestions. This allows suggestions to be made at the appropriate time by adjusting the timing according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the suggestion unit may be performed using AI or not. For example, the suggestion unit can input user emotion data into a generative AI and have the generative AI adjust the timing of suggestions.

[0121] The suggestion unit selects the optimal suggestion method when making a suggestion, taking into account the user's device information. For example, if the user is using a smartphone, the suggestion unit provides a suggestion method optimized for smartphones. If the user is using a tablet, the suggestion unit provides a suggestion method optimized for tablets. If the user is using a smartwatch, the suggestion unit provides a suggestion method optimized for smartwatches. By selecting the optimal suggestion method while considering the user's device information, it becomes possible to make the best possible suggestions for the user. Some or all of the above processing in the suggestion unit may be performed using AI, or it may be performed without AI. For example, the suggestion unit can input the user's device information into a generating AI and have the generating AI select the optimal suggestion method.

[0122] The proposal unit adjusts the frequency of proposals, taking into account the expiration dates of point campaigns. For example, the proposal unit prioritizes proposing campaigns with approaching expiration dates. It reduces the frequency of proposals for campaigns with longer expiration dates. The proposal unit makes proposals at the optimal time according to the user's schedule. By adjusting the frequency of proposals, taking into account the expiration dates of point campaigns, proposals can be made at the appropriate time. Some or all of the above processing in the proposal unit may be performed using AI or not. For example, the proposal unit can input data on the expiration dates of point campaigns into a generation AI and have the generation AI adjust the frequency of proposals.

[0123] The AR display unit estimates the user's emotions and adjusts the content of the AR display based on the estimated emotions. For example, if the user is stressed, the AR display unit will display simple and highly visible content. If the user is relaxed, the AR display unit will display content that includes detailed information. If the user is in a hurry, the AR display unit will include only essential information in the AR display content. In this way, by adjusting the content of the AR display according to the user's emotions, the optimal display content can be provided to the user. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input user emotion data into a generative AI and have the generative AI perform the adjustment of the AR display content.

[0124] The AR display unit determines the display priority based on the importance of locations and products where points can be earned during AR display. For example, the AR display unit may prioritize displaying locations and products that offer high points in AR. The AR display unit may also prioritize displaying locations and products where points can be earned for a limited time in AR. The AR display unit determines the display priority based on the importance of the user's interests. This allows important information to be displayed preferentially by determining the display priority based on the importance of locations and products where points can be earned. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input data on the importance of locations and products where points can be earned into a generating AI and have the generating AI perform the determination of the display priority.

[0125] The AR display unit selects the optimal display method by referring to the user's past behavior patterns when displaying AR content. For example, the AR display unit selects the optimal AR display method based on the point acquisition methods the user has used in the past. The AR display unit analyzes the user's past behavior patterns and selects the most efficient AR display method. The AR display unit selects the optimal AR display method in real time based on the user's past behavior patterns. This makes it possible to display content optimally for the user by selecting the optimal display method by referring to the user's past behavior patterns. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input data on the user's past behavior patterns into a generating AI and have the generating AI perform the selection of the optimal display method.

[0126] The AR display unit estimates the user's emotions and adjusts the timing of AR displays based on the estimated emotions. For example, if the user is stressed, the AR display unit reduces the frequency of AR displays. If the user is relaxed, the AR display unit maintains the normal frequency of AR displays. If the user is in a hurry, the AR display unit prioritizes only important AR displays. This allows for displays to be shown at the appropriate time by adjusting the timing of AR displays according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the AR display unit may be performed using AI or not. For example, the AR display unit can input user emotion data into a generative AI and have the generative AI adjust the timing of AR displays.

[0127] The AR display unit selects the optimal display method when displaying AR content, taking into account the user's device information. For example, if the user is using a smartphone, the AR display unit provides an AR display method optimized for smartphones. If the user is using a tablet, the AR display unit provides an AR display method optimized for tablets. If the user is using a smartwatch, the AR display unit provides an AR display method optimized for smartwatches. By selecting the optimal display method considering the user's device information, the optimal display for the user becomes possible. Some or all of the above processing in the AR display unit may be performed using AI, or it may be performed without using AI. For example, the AR display unit can input the user's device information into a generating AI and have the generating AI select the optimal display method.

[0128] The AR display unit adjusts the display frequency when displaying AR content, taking into account the expiration dates of point campaigns. For example, the AR display unit prioritizes displaying campaigns with approaching expiration dates. The AR display unit reduces the display frequency of campaigns with longer expiration dates. The AR display unit displays AR content at the optimal timing according to the user's schedule. By adjusting the display frequency while considering the expiration dates of point campaigns, the display can be performed at the appropriate time. Some or all of the above processing in the AR display unit may be performed using AI, or not. For example, the AR display unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the adjustment of the display frequency.

[0129] The learning unit estimates the user's emotions and selects training data based on the estimated emotions. For example, if the user is stressed, the learning unit prioritizes training data that helps reduce stress. If the user is relaxed, the learning unit uses normal training data. If the user is in a hurry, the learning unit prioritizes data that allows for rapid learning. This allows for optimal learning for the user by selecting training data according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input user emotion data into a generative AI and have the generative AI perform the selection of training data.

[0130] The learning unit optimizes the learning algorithm by referring to past learning data during the learning process. For example, the learning unit analyzes past learning data and selects the optimal learning algorithm. The learning unit extracts effective learning patterns from past learning data. The learning unit optimizes the learning algorithm in real time based on past learning data. This improves the accuracy of learning by optimizing the learning algorithm by referring to past learning data. Some or all of the above processes in the learning unit may be performed using AI or not. For example, the learning unit can input past learning data into a generating AI and have the generating AI perform the optimization of the learning algorithm.

[0131] The learning unit determines learning priorities based on the user's behavior patterns during the learning process. For example, the learning unit prioritizes learning point-earning methods that the user frequently uses. The learning unit analyzes the user's behavior patterns and determines the most effective learning order. The learning unit adjusts learning priorities in real time based on the user's past behavior patterns. This enables efficient learning by determining learning priorities based on the user's behavior patterns. Some or all of the above processes in the learning unit may be performed using AI or not. For example, the learning unit can input user behavior pattern data into a generating AI and have the generating AI perform the determination of learning priorities.

[0132] The learning unit estimates the user's emotions and adjusts the learning frequency based on the estimated emotions. For example, if the user is stressed, the learning unit reduces the learning frequency. If the user is relaxed, the learning unit maintains the normal learning frequency. If the user is in a hurry, the learning unit prioritizes only important learning. This allows learning to be performed at an appropriate frequency by adjusting the learning frequency according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input user emotion data into a generative AI and have the generative AI adjust the learning frequency.

[0133] The learning unit selects the optimal learning method during learning, taking into account the user's device information. For example, if the user is using a smartphone, the learning unit provides a learning method optimized for smartphones. If the user is using a tablet, the learning unit provides a learning method optimized for tablets. If the user is using a smartwatch, the learning unit provides a learning method optimized for smartwatches. By selecting the optimal learning method while considering the user's device information, the learning unit enables optimal learning for the user. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input the user's device information into a generating AI and have the generating AI select the optimal learning method.

[0134] The learning unit weights the training data during training, taking into account the expiration dates of point campaigns. For example, the learning unit gives higher weight to training data for campaigns with approaching expiration dates, and lower weight to training data for campaigns with longer expiration dates. The learning unit adjusts the weighting of the training data in real time according to the user's schedule. This allows training to be performed at the appropriate time by weighting the training data while considering the expiration dates of point campaigns. Some or all of the above processing in the learning unit may be performed using AI or not. For example, the learning unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI perform the weighting of the training data.

[0135] The calculation unit estimates the user's emotions and adjusts the timing of point calculations based on the estimated emotions. For example, if the user is stressed, the calculation unit reduces the frequency of point calculations. If the user is relaxed, the calculation unit maintains the normal frequency of point calculations. If the user is in a hurry, the calculation unit prioritizes only important point calculations. This allows calculations to be performed at the appropriate time by adjusting the timing of point calculations according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input user emotion data into a generative AI and have the generative AI adjust the timing of point calculations.

[0136] The calculation unit determines the calculation priority based on the importance of the points earned when calculating points. For example, the calculation unit may prioritize the acquisition of high points. The calculation unit may prioritize the acquisition of points that are available for a limited time. The calculation unit determines the importance and calculation priority based on the user's interests. In this way, important points can be calculated preferentially by determining the calculation priority based on the importance of the points earned. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input data on the importance of the points earned into a generating AI and have the generating AI perform the determination of the calculation priority.

[0137] The calculation unit selects the optimal calculation method by referring to the user's past point history when calculating points. For example, the calculation unit selects the optimal calculation method based on the user's past point history. The calculation unit analyzes the user's past point history and selects the most efficient calculation method. The calculation unit optimizes the calculation method in real time based on the user's past point history. This enables efficient point calculation by selecting the optimal calculation method by referring to the user's past point history. Some or all of the above processes in the calculation unit may be performed using AI or not. For example, the calculation unit can input data of the user's past point history into a generating AI and have the generating AI perform the selection of the optimal calculation method.

[0138] The calculation unit estimates the user's emotions and customizes the point calculation based on the estimated emotions. For example, if the user is stressed, the calculation unit will create a concise and to-the-point point calculation. If the user is relaxed, the calculation unit will create a point calculation that includes detailed information. If the user is in a hurry, the calculation unit will include only the essential information in the point calculation. In this way, by customizing the point calculation according to the user's emotions, the system can provide the user with the most optimal calculation. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input user emotion data into a generative AI and have the generative AI perform the customization of the point calculation.

[0139] The calculation unit selects the optimal calculation method when calculating points, taking into account the user's device information. For example, if the user is using a smartphone, the calculation unit provides a calculation method optimized for smartphones. If the user is using a tablet, the calculation unit provides a calculation method optimized for tablets. If the user is using a smartwatch, the calculation unit provides a calculation method optimized for smartwatches. By selecting the optimal calculation method while considering the user's device information, the calculation unit enables the user to perform calculations that are optimal for them. Some or all of the above processing in the calculation unit may be performed using AI, or it may be performed without AI. For example, the calculation unit can input the user's device information into a generating AI and have the generating AI perform the selection of the optimal calculation method.

[0140] The calculation unit adjusts the frequency of point calculations, taking into account the expiration dates of point campaigns. For example, the calculation unit prioritizes calculating points for campaigns with approaching expiration dates. It reduces the frequency of point calculations for campaigns with longer expiration dates. The calculation unit performs point calculations at the optimal time according to the user's schedule. By adjusting the frequency of calculations, taking into account the expiration dates of point campaigns, calculations can be performed at the appropriate time. Some or all of the above processing in the calculation unit may be performed using AI or not. For example, the calculation unit can input data on the expiration dates of point campaigns into a generating AI and have the generating AI adjust the calculation frequency.

[0141] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0142] The proposal department can acquire users' health data and suggest ways to earn points based on their health status. For example, it can acquire users' step count data and propose a campaign where users can earn points when they reach a certain number of steps. It can also acquire users' heart rate data and propose ways to earn points when they engage in moderate exercise. Furthermore, it can acquire users' sleep data and propose a campaign where users can earn points when they get enough sleep. In this way, by suggesting ways to earn points that are tailored to the user's health status, it is possible to promote health.

[0143] The suggestion function can estimate the user's emotions and propose ways to earn points based on those emotions. For example, if the user is feeling stressed, it can suggest ways to earn points related to relaxing places or activities. If the user is happy, it can also suggest ways to earn points related to entertainment or leisure that will further enhance that feeling. Furthermore, if the user is tired, it can suggest ways to earn points related to rest or refreshment. In this way, by suggesting ways to earn points that match the user's emotions, user satisfaction can be increased.

[0144] The proposal department can suggest ways for users to earn points based on their hobbies and interests. For example, if a user is interested in music, it can suggest ways to earn points related to music-related events and products. If a user is interested in sports, it can suggest ways to earn points related to sports events and sporting goods. Furthermore, if a user is interested in cooking, it can suggest ways to earn points related to cooking classes and ingredients. In this way, by suggesting ways to earn points that match the user's hobbies and interests, it is possible to attract the user's interest.

[0145] The suggestion function can propose ways for users to earn points based on their past purchase history. For example, it can suggest point campaigns related to products the user has purchased in the past. It can also suggest ways to earn points related to stores or brands that the user frequently uses. Furthermore, if a user tends to purchase products in a particular category, it can suggest ways to earn points related to that category. In this way, by suggesting ways to earn points based on the user's purchase history, it is possible to increase the user's motivation to purchase.

[0146] The proposal team can suggest ways for users to earn points based on their social media activity. For example, they can propose campaigns where users earn points when they share posts about specific brands or products. They can also suggest ways for users to earn points when they post using specific hashtags. Furthermore, they can suggest ways for users to earn points when they participate in specific events or campaigns. By suggesting point-earning methods based on users' social media activity, this can increase user engagement.

[0147] The suggestion function can estimate the user's emotions and propose ways to earn points based on those emotions. For example, if the user is feeling stressed, it can suggest ways to earn points related to relaxing places or activities. If the user is happy, it can also suggest ways to earn points related to entertainment or leisure that will further enhance that feeling. Furthermore, if the user is tired, it can suggest ways to earn points related to rest or refreshment. In this way, by suggesting ways to earn points that match the user's emotions, user satisfaction can be increased.

[0148] The proposal team can suggest ways for users to earn points based on their life events. For example, they can propose a special points campaign on a user's birthday. They can also suggest ways to earn points related to special days such as a user's wedding anniversary or a child's birthday. Furthermore, they can suggest ways to earn points related to life events such as moving or starting a new job. By suggesting ways to earn points that are tailored to a user's life events, they can make special days even more special for the user.

[0149] The suggestion function can estimate the user's emotions and propose ways to earn points based on those emotions. For example, if the user is feeling stressed, it can suggest ways to earn points related to relaxing places or activities. If the user is happy, it can also suggest ways to earn points related to entertainment or leisure that will further enhance that feeling. Furthermore, if the user is tired, it can suggest ways to earn points related to rest or refreshment. In this way, by suggesting ways to earn points that match the user's emotions, user satisfaction can be increased.

[0150] The proposal department can suggest ways for users to earn points based on their travel plans. For example, it can suggest point campaigns that users can use at their travel destination. It can also suggest ways to earn points related to tourist attractions and restaurants that users will visit during their trip. Furthermore, it can suggest ways to earn points related to items that users need to prepare before their trip. In this way, by suggesting ways to earn points that match the user's travel plans, the enjoyment of their trip can be further enhanced.

[0151] The suggestion function can estimate the user's emotions and propose ways to earn points based on those emotions. For example, if the user is feeling stressed, it can suggest ways to earn points related to relaxing places or activities. If the user is happy, it can also suggest ways to earn points related to entertainment or leisure that will further enhance that feeling. Furthermore, if the user is tired, it can suggest ways to earn points related to rest or refreshment. In this way, by suggesting ways to earn points that match the user's emotions, user satisfaction can be increased.

[0152] The following briefly describes the processing flow for example form 2.

[0153] Step 1: The recognition unit recognizes products and advertisements captured by the camera. The recognition unit uses AI to analyze images of products and advertisements and extract specific features. For example, it recognizes logos and text on products and advertisements and notifies users of point campaigns based on that recognition. Step 2: The notification unit notifies users of point campaigns based on the products and advertisements recognized by the recognition unit. The notification unit uses AI to send notifications at the optimal time based on the user's interests and behavioral patterns. For example, if a user shows interest in a particular product, it will notify them of point campaigns related to that product. Step 3: The voice recognition unit recognizes in-store announcements and conversations. The voice recognition unit uses AI to analyze the voice data and extract specific keywords and phrases. For example, it can detect special sale information and campaigns from in-store announcements and conversations. Step 4: The voice notification unit notifies the user of special offers and campaigns recognized by the voice recognition unit. The voice notification unit uses AI to provide optimal voice notifications based on the user's auditory preferences. For example, if the user prefers a particular voice notification, that notification will be prioritized. Step 5: The proposal team uses GPS data to suggest point-earning methods based on the user's current location. The proposal team uses AI to analyze the user's location information and propose the optimal point-earning method. For example, if the user is in a specific area, it will suggest point-earning methods available in that area.

[0154] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0155] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0156] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0157] Each of the multiple elements described above, including the recognition unit, notification unit, voice recognition unit, voice notification unit, suggestion unit, AR display unit, learning unit, and calculation unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the recognition unit uses the camera 42 of the smart device 14 to recognize products and advertisements, and the data is processed by the control unit 46A. The notification unit notifies the user of point campaigns using the control unit 46A of the smart device 14. The voice recognition unit uses the microphone 38B of the smart device 14 to recognize in-store announcements and conversations, and the data is processed by the control unit 46A. The voice notification unit uses the speaker 40B of the smart device 14 to notify the user of opportunities to earn points. The suggestion unit uses GPS data from the smart device 14 to suggest ways to earn points based on the user's current location. The AR display unit uses the camera 42 of the smart device 14 to overlay information on how to earn points onto the view the user has seen. The learning unit uses the identification processing unit 290 of the data processing unit 12 to learn the user's past behavior patterns and preferences. The calculation unit calculates the points acquired by the specific processing unit 290 of the data processing device 12 in real time and notifies the control unit 46A of the smart device 14. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0158] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0159] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0160] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0161] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0162] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0163] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0164] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0165] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0166] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0167] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0168] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0169] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0170] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0171] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0172] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0173] Each of the multiple elements described above, including the recognition unit, notification unit, voice recognition unit, voice notification unit, suggestion unit, AR display unit, learning unit, and calculation unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the recognition unit uses the camera 42 of the smart glasses 214 to recognize products and advertisements, and the data is processed by the control unit 46A. The notification unit notifies the user of point campaigns using the control unit 46A of the smart glasses 214. The voice recognition unit uses the microphone 238 of the smart glasses 214 to recognize in-store announcements and conversations, and the data is processed by the control unit 46A. The voice notification unit uses the speaker 240 of the smart glasses 214 to notify the user of opportunities to earn points. The suggestion unit uses GPS data from the smart glasses 214 to suggest ways to earn points based on the user's current location. The AR display unit uses the camera 42 of the smart glasses 214 to overlay information on how to earn points onto the view the user has seen. The learning unit uses the identification processing unit 290 of the data processing unit 12 to learn the user's past behavior patterns and preferences. The calculation unit calculates the points acquired by the specific processing unit 290 of the data processing device 12 in real time and notifies the control unit 46A of the smart glasses 214. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0174] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0175] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0176] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0177] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0178] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0179] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0180] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0181] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0182] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0183] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0184] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0185] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0186] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0187] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0188] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0189] Each of the multiple elements described above, including the recognition unit, notification unit, voice recognition unit, voice notification unit, suggestion unit, AR display unit, learning unit, and calculation unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the recognition unit uses the camera 42 of the headset terminal 314 to recognize products and advertisements, and processes them using the control unit 46A. The notification unit notifies the user of point campaigns using the control unit 46A of the headset terminal 314. The voice recognition unit uses the microphone 238 of the headset terminal 314 to recognize in-store announcements and conversations, and processes them using the control unit 46A. The voice notification unit uses the speaker 240 of the headset terminal 314 to notify the user of point earning opportunities. The suggestion unit uses GPS data from the headset terminal 314 to suggest ways to earn points based on the user's current location. The AR display unit overlays point earning information onto the view seen using the camera 42 of the headset terminal 314. The learning unit learns the user's past behavior patterns and preferences using the specific processing unit 290 of the data processing device 12. The calculation unit calculates the points acquired by the specific processing unit 290 of the data processing device 12 in real time and notifies the user via the control unit 46A of the headset terminal 314. The correspondence between each unit and the device and control unit is not limited to the example described above and can be modified in various ways.

[0190] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0191] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0192] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0193] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0194] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0195] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0196] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0197] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0198] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0199] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0200] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0201] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0202] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0203] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0204] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0205] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0206] Each of the multiple elements described above, including the recognition unit, notification unit, voice recognition unit, voice notification unit, suggestion unit, AR display unit, learning unit, and calculation unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the recognition unit uses the camera 42 of the robot 414 to recognize products and advertisements, and the data is processed by the control unit 46A. The notification unit notifies the user of point campaigns via the control unit 46A of the robot 414. The voice recognition unit uses the microphone 238 of the robot 414 to recognize in-store announcements and conversations, and the data is processed by the control unit 46A. The voice notification unit uses the speaker 240 of the robot 414 to notify the user of opportunities to earn points. The suggestion unit uses the GPS data of the robot 414 to suggest ways to earn points based on the user's current location. The AR display unit overlays information on how to earn points onto the view seen using the camera 42 of the robot 414. The learning unit learns the user's past behavior patterns and preferences via the identification processing unit 290 of the data processing unit 12. The calculation unit calculates the points acquired by the specific processing unit 290 of the data processing device 12 in real time and notifies the control unit 46A of the robot 414. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0207] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0208] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0209] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0210] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0211] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0212] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0213] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0214] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0215] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0216] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0217] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0218] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0219] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0220] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0221] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0222] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0223] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0224] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0225] (Note 1) A recognition unit that recognizes products and advertisements captured by the camera, A notification unit that notifies a point campaign based on the products and advertisements recognized by the recognition unit, A voice recognition unit that recognizes in-store announcements and conversations, A voice notification unit that notifies the user of special sale information and campaigns recognized by the voice recognition unit, The proposal department proposes a method for earning points based on the user's current location using GPS data, Equipped with A system characterized by the following features. (Note 2) It features an AR display unit that overlays locations and products where points can be earned onto the actual scenery viewed through the camera. The system described in Appendix 1, characterized by the features described herein. (Note 3) It features a learning unit that learns the user's past behavior patterns and preferences and proposes point-earning actions optimized for that individual. The system described in Appendix 1, characterized by the features described herein. (Note 4) It includes a calculation unit that calculates earned points in real time and reports them via voice or push notification. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned recognition unit, Instantly recognizes products and advertisements captured by the camera. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned speech recognition unit, Detecting special sale information and campaigns from in-store announcements and conversations. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned recognition unit, It estimates the user's emotions and adjusts the recognition accuracy based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 8) The aforementioned recognition unit, During recognition, different recognition algorithms are applied depending on the product's brand and category. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned recognition unit, During recognition, the recognition priority is determined based on the frequency of use and popularity of the product. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned recognition unit, It estimates the user's emotions and adjusts how the recognition results are displayed based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned recognition unit, During recognition, the recognition algorithm is optimized based on the color and shape of the product. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned recognition unit, During recognition, relevant product information is acquired in real time and reflected in the recognition result. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned notification unit, It estimates the user's emotions and adjusts the timing of notifications based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned notification unit, When sending notifications, we prioritize them based on the importance of the points campaign. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned notification unit, When sending a notification, the system will refer to the user's past notification history to select the most suitable notification method. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned notification unit, It estimates the user's emotions and customizes the content of notifications based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned notification unit, When sending notifications, the system selects the most suitable notification method, taking into account the user's device information. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned notification unit, When sending notifications, the frequency of notifications will be adjusted to take into account the expiration date of the points campaign. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned speech recognition unit, It estimates the user's emotions and adjusts the accuracy of speech recognition based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned speech recognition unit, During voice recognition, the system prioritizes recognition based on the importance of special offers and campaigns. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned speech recognition unit, During voice recognition, the recognition algorithm is optimized according to the noise level inside the store. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned speech recognition unit, It estimates the user's emotions and adjusts how the speech recognition results are displayed based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned speech recognition unit, When using voice recognition, different recognition algorithms are applied depending on the category of the sale information or campaign. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned speech recognition unit, During voice recognition, the system analyzes the store's acoustic environment in real time to improve recognition accuracy. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned voice notification unit, It estimates the user's emotions and adjusts the timing of voice notifications based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned voice notification unit, When an audio notification is sent, the system prioritizes notifications based on the importance of special offers and campaigns. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned voice notification unit, When an audio notification is sent, the system will refer to the user's past notification history to select the most suitable notification method. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned voice notification unit, It estimates the user's emotions and customizes the content of voice notifications based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned voice notification unit, When an audio notification is sent, the system selects the optimal notification method by considering the user's device information. The system described in Appendix 1, characterized by the features described herein. (Note 30) The aforementioned voice notification unit, When sending voice notifications, the frequency of notifications will be adjusted to take into account the expiration dates of special offers and campaigns. The system described in Appendix 1, characterized by the features described herein. (Note 31) The aforementioned proposal section is, It estimates the user's emotions and adjusts the content of the suggestions based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 32) The aforementioned proposal section is, When making a proposal, the optimal method for earning points is selected based on the user's current location. The system described in Appendix 1, characterized by the features described herein. (Note 33) The aforementioned proposal section is, When making a proposal, we refer to the user's past behavior patterns to make the most appropriate suggestion. The system described in Appendix 1, characterized by the features described herein. (Note 34) The aforementioned proposal section is, It estimates the user's emotions and adjusts the timing of suggestions based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 35) The aforementioned proposal section is, When making a proposal, the optimal proposal method is selected considering the user's device information. The system described in Appendix 1, characterized by the features described herein. (Note 36) The aforementioned proposal section is, When making a proposal, adjust the frequency of the proposal, taking into account the expiration date of the points campaign. The system described in Appendix 1, characterized by the features described herein. (Note 37) The AR display unit is It estimates the user's emotions and adjusts the AR display content based on the estimated emotions. The system described in Appendix 2, characterized by the features described herein. (Note 38) The AR display unit is When displaying AR content, the display priority is determined based on the importance of locations and products where points can be earned. The system described in Appendix 2, characterized by the features described herein. (Note 39) The AR display unit is When displaying AR content, the system selects the optimal display method by referring to the user's past behavior patterns. The system described in Appendix 2, characterized by the features described herein. (Note 40) The AR display unit is It estimates the user's emotions and adjusts the timing of AR displays based on those emotions. The system described in Appendix 2, characterized by the features described herein. (Note 41) The AR display unit is When performing AR display, select the optimal display method considering the user's device information The system according to Appendix 2, characterized by this (Appendix 42) The AR display unit When performing AR display, adjust the display frequency considering the expiration date of the point campaign The system according to Appendix 2, characterized by this (Appendix 43) The learning unit Estimate the user's emotion and select learning data based on the estimated user emotion The system according to Appendix 3, characterized by this (Appendix 44) The learning unit When learning, optimize the learning algorithm by referring to past learning data The system according to Appendix 3, characterized by this ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​The system estimates the user's emotions and adjusts the timing of point calculations based on those emotions. The system described in Appendix 4, characterized by the features described herein. (Note 50) The calculation unit, When calculating points, the priority of the calculation is determined based on the importance of the points earned. The system described in Appendix 4, characterized by the features described herein. (Note 51) The calculation unit, When calculating points, the system selects the optimal calculation method by referring to the user's past point history. The system described in Appendix 4, characterized by the features described herein. (Note 52) The calculation unit, It estimates the user's emotions and customizes the point calculation based on those estimated emotions. The system described in Appendix 4, characterized by the features described herein. (Note 53) The calculation unit, When calculating points, the optimal calculation method is selected by taking into account the user's device information. The system described in Appendix 4, characterized by the features described herein. (Note 54) The calculation unit, When calculating points, the calculation frequency is adjusted to take into account the expiration date of the point campaign. The system described in Appendix 4, characterized by the features described herein. [Explanation of symbols]

[0226] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A recognition unit that recognizes products and advertisements captured by the camera, A notification unit that notifies a point campaign based on the products and advertisements recognized by the recognition unit, A voice recognition unit that recognizes in-store announcements and conversations, A voice notification unit that notifies the user of special sale information and campaigns recognized by the voice recognition unit, The proposal department proposes a method for earning points based on the user's current location using GPS data, Equipped with A system characterized by the following features.

2. It features an AR display unit that overlays locations and products where points can be earned onto the actual scenery viewed through the camera. The system according to feature 1.

3. It features a learning unit that learns the user's past behavior patterns and preferences and proposes point-earning actions optimized for that individual. The system according to feature 1.

4. It includes a calculation unit that calculates earned points in real time and reports them via voice or push notification. The system according to feature 1.

5. The aforementioned recognition unit, Instantly recognizes products and advertisements captured by the camera. The system according to feature 1.

6. The aforementioned speech recognition unit, Detecting special sale information and campaigns from in-store announcements and conversations. The system according to feature 1.

7. The aforementioned recognition unit, It estimates the user's emotions and adjusts the recognition accuracy based on the estimated emotions. The system according to feature 1.

8. The aforementioned recognition unit, During recognition, different recognition algorithms are applied depending on the product's brand and category. The system according to feature 1.

9. The aforementioned recognition unit, During recognition, the recognition priority is determined based on the frequency of use and popularity of the product. The system according to feature 1.

10. The aforementioned recognition unit, It estimates the user's emotions and adjusts how the recognition results are displayed based on the estimated emotions. The system according to feature 1.

Images