system

Abstract

The system according to this embodiment aims to make self-checkout operations user-friendly. [Solution] The system according to the embodiment comprises an identification unit, an addition unit, a support unit, and an analysis unit. The identification unit recognizes the customer's face. The addition unit adds points based on the customer information recognized by the identification unit. The support unit supports the operation of the self-checkout register. The analysis unit analyzes the operation data of the self-checkout register.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, since the operation of self-checkout is not user-friendly, there is a risk that users may have difficulty in operating it.

[0005] The system according to the embodiment aims to make the operation of self-checkout user-friendly.

Means for Solving the Problems

[0006] The system according to the embodiment includes an identification unit, an addition unit, a support unit, and an analysis unit. The identification unit recognizes the face of a customer. The addition unit adds points based on the information of the customer recognized by the identification unit. The support unit supports the operation of self-checkout. The analysis unit analyzes the operation data of self-checkout.

Effects of the Invention

[0007] The system according to this embodiment can make self-checkout operations user-friendly. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the reception device 38, the output device 40, and the camera 42 are connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The self-checkout system according to an embodiment of the present invention is a new self-checkout system that utilizes facial recognition technology and generative AI to address the labor shortage in Japan's service industry, amidst the spread of self-checkout systems during the COVID-19 pandemic. This self-checkout system recognizes the face of a customer entering the store, and if the customer is registered as a member, it can use facial recognition to add points and make payments without a card. Generative AI supports the operation of the self-checkout system, enabling user-friendly operation. This promotes efficient checkout operations, compensates for the labor shortage in stores, and increases labor productivity per person. As a result, the self-checkout system can significantly improve customer convenience. For example, a system that recognizes the face of a customer entering the store is activated, and facial recognition technology is used to identify the customer's face. If the customer is registered as a member, points can be added and payments can be made without a card. This significantly improves customer convenience. Next, generative AI supports the operation of the self-checkout system. For example, if a customer is confused about scanning products or processing payments, the generative AI provides support in real time, enabling user-friendly operation. This eliminates the disadvantages of self-checkout systems, allowing customers to use them smoothly. Furthermore, the generating AI analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. For example, it can suggest optimal checkout placement to reduce checkout wait times during peak hours and improvements to increase product scanning speed. This improves store operational efficiency and helps compensate for labor shortages. This system will bring about a major transformation in the Japanese retail industry and provide new consumer experiences. For example, customers can enjoy a smooth and efficient shopping experience by using self-checkouts with the support of facial recognition and generating AI. Stores can also increase labor productivity per person by achieving efficient checkout operations and compensating for labor shortages.

[0029] The self-checkout system according to this embodiment comprises an identification unit, an addition unit, a support unit, and an analysis unit. The identification unit recognizes the customer's face. The identification unit identifies the customer's face using, for example, a face recognition algorithm. The identification unit can use, for example, a camera as the sensor. The addition unit adds points based on the customer information recognized by the identification unit. The addition unit sets the unit of points and determines the timing of the addition. The addition unit may include AI processing. The support unit supports the operation of the self-checkout system. The support unit uses generative AI to provide real-time support when the customer is confused about scanning products or processing payments. The support unit provides, for example, voice guidance or on-screen guidance. The analysis unit analyzes the operation data of the self-checkout system. The analysis unit makes suggestions to improve the operational efficiency of the store using, for example, data collection methods and analysis algorithms. As a result, the self-checkout system according to this embodiment is capable of customer face recognition, point addition, self-checkout operation support, and operation data analysis.

[0030] The identification unit recognizes the customer's face. For example, the identification unit identifies the customer's face using a facial recognition algorithm. Specifically, the identification unit uses a high-resolution camera to acquire a facial image of the customer and inputs it into the facial recognition algorithm. The facial recognition algorithm uses a model pre-trained with deep learning technology to extract the customer's facial features. These features include information such as the position of the eyes, the shape of the nose, and the contour of the mouth. Based on these features, the identification unit matches the customer's face against existing facial data in the database to identify the customer. If the match is successful, the identification unit obtains the customer's ID information and passes it on to the next process. To improve the matching accuracy, the identification unit can also perform correction processing that takes into account environmental factors such as lighting conditions and camera angle. Furthermore, the identification unit can achieve more accurate facial recognition by using multiple cameras to acquire facial images from different angles and generating a 3D model. This allows the identification unit to quickly and accurately recognize customer faces and provide efficient service in conjunction with other functions of the self-checkout system.

[0031] The point accrual unit adds points based on customer information recognized by the identification unit. Specifically, the accrual unit receives customer ID information and accesses the customer's points account. The accrual unit sets the unit of points and calculates points based on, for example, the purchase amount. A system can also be implemented where bonus points are added when the purchase amount exceeds a certain threshold. The accrual unit can also use AI to analyze the customer's purchase history and offer additional points for specific products or campaigns. For example, customers who purchase specific products during a specific period can be awarded bonus points in addition to regular points. The accrual unit processes point accrual in real time, ensuring that points are reflected immediately when the customer passes through the checkout. Furthermore, the accrual unit records the point accrual history so that customers can review it later. This allows the accrual unit to provide a transparent points service to customers and improve customer satisfaction.

[0032] The support department assists with the operation of self-checkout machines. Specifically, the support department uses generative AI to provide real-time support when customers encounter difficulties scanning items or processing payments. The generative AI uses natural language processing technology to understand customer questions and problems and generates appropriate answers and instructions. For example, if a customer is unable to scan an item's barcode, the support department will provide voice guidance such as, "Please bring the barcode closer to the scanner." It can also visually display scanning procedures and points to note through on-screen displays. The support department monitors customer operation history in real time and intervenes immediately to provide support when problems occur. Furthermore, the support department can provide personalized support based on the customer's past operation and purchase history. For example, if a particular customer has repeatedly encountered the same problem in the past, special attention can be paid to that customer to prevent the problem from recurring. In this way, the support department can enable customers to use self-checkout machines smoothly and improve the customer experience.

[0033] The analytics department analyzes operation data from self-checkout registers. Specifically, the analytics department collects customer operation and purchase data and uses this data to propose ways to improve store operational efficiency. The analytics department uses self-checkout operation logs, customer purchase history, and point accumulation history as data collection methods. This data is integrated and analyzed using AI. For example, it identifies which operations customers spend the most time on at self-checkout registers and analyzes the causes. It can extract specific problems, such as difficulty scanning certain products or delays in payment processing. The analytics department proposes solutions to these problems, improving store operational efficiency. For example, for products that are difficult to scan, measures such as changing the barcode position can be taken. The analytics department can also analyze customer purchasing patterns and predict congestion levels at specific times of day and on specific days of the week. This allows stores to appropriately allocate staff and manage inventory. Furthermore, the analytics department collects customer feedback and provides data to continuously improve the usability and convenience of self-checkout registers. In this way, the analytics department can not only improve store operational efficiency but also contribute to increased customer satisfaction.

[0034] The support unit can assist customers in real time using generative AI. For example, the support unit can use generative AI to provide real-time support when customers are confused about scanning products or processing payments. The support unit can also use generative AI to provide voice guidance and on-screen instructions. This makes self-checkout operations smoother by assisting customers in real time using generative AI. Some or all of the above processes in the support unit may be performed using generative AI or not. For example, the support unit can use an AI model that uses generative AI to assist customers in real time and support customer operations.

[0035] The analysis unit may include a proposal unit that analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. For example, the analysis unit makes suggestions to improve store operational efficiency using data collection methods and analysis algorithms. The proposal unit can, for example, suggest optimal checkout layouts to reduce checkout waiting times during peak hours, or suggest improvements to increase product scanning speed. This enables suggestions to improve store operational efficiency. Some or all of the above processing in the analysis unit may be performed using AI, or not. For example, the analysis unit can input self-checkout operation data into an AI model and have the AI ​​model execute suggestions to improve store operational efficiency.

[0036] The identification unit recognizes the customer's face and, if the customer is registered, can add points or process payments without a card. The identification unit identifies the customer's face using, for example, a facial recognition algorithm. The identification unit can use, for example, a camera as a sensor. The identification unit can add points or process payments without a card if the customer is registered. This allows for the addition of points and payment without a card based on the customer's facial recognition. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input the customer's facial recognition data into an AI model and have the AI ​​model perform the addition of points or payment.

[0037] The addition unit can add points based on customer facial recognition. The addition unit authenticates the customer's face using, for example, a facial recognition algorithm. The addition unit can use, for example, a camera as the sensor. The addition unit adds points based on customer facial recognition. This allows points to be added based on customer facial recognition. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer facial recognition data into an AI model and have the AI ​​model perform the point addition.

[0038] The identification unit can analyze a customer's past visit history and select the optimal recognition method. For example, if a customer visits frequently, the identification unit can improve the accuracy of facial recognition to recognize them quickly. If a customer is visiting for the first time, the identification unit can maintain normal facial recognition accuracy and perform detailed recognition. If a customer visits during a specific time period, the identification unit can select the optimal recognition method for that time period. This improves recognition accuracy by selecting the optimal recognition method based on the customer's past visit history. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer visit history data into an AI model and have the AI ​​model select the optimal recognition method.

[0039] The identification unit can determine the recognition priority based on the customer's current purchase intent when recognizing a customer's face. For example, if the customer shows high purchase intent, the identification unit will increase the priority of face recognition to recognize them quickly. If the customer shows low purchase intent, the identification unit will maintain the normal priority of face recognition. If the customer shows interest in a specific product, the identification unit can prioritize recognition related to that product. This enables rapid recognition by determining the recognition priority based on the customer's purchase intent. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer purchase intent data into an AI model and have the AI ​​model execute the recognition priority.

[0040] The identification unit can optimize the recognition results by taking into account the customer's geographical location information. For example, if the customer is in a specific region, the identification unit can prioritize displaying information related to that region. If the customer is on the move, the identification unit can display information based on their current location in real time. If the customer is in a specific store, the identification unit can prioritize displaying information related to that store. In this way, by optimizing the recognition results based on the customer's geographical location information, highly relevant information can be provided. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input the customer's geographical location data into an AI model and have the AI ​​model perform the optimization of the recognition results.

[0041] The identification unit can analyze a customer's social media activity and provide relevant recognition results. For example, if a customer mentions a specific product on social media, the identification unit can display information related to that product. If a customer mentions a specific store on social media, the identification unit can display information related to that store. If a customer mentions a specific event on social media, the identification unit can display information related to that event. In this way, by providing relevant recognition results based on the customer's social media activity, information tailored to the customer's interests can be displayed. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer social media activity data into an AI model and have the AI ​​model perform the task of providing relevant recognition results.

[0042] The point accrual unit can analyze a customer's past point usage history and select the optimal accrual method. For example, if a customer uses points frequently, the accrual unit can increase the frequency of point accrual. If a customer uses points at a specific time, the accrual unit can accrue points at that time. If a customer uses points for a specific product, the accrual unit can select a point accrual method related to that product. This improves the efficiency of point accrual by selecting the optimal accrual method based on the customer's past point usage history. Some or all of the above processing in the accrual unit may be performed using AI or not. For example, the accrual unit can input customer point usage history data into an AI model and have the AI ​​model select the optimal accrual method.

[0043] The point accrual unit can determine the priority of point accrual based on the customer's current purchase intent. For example, if the customer shows high purchase intent, the point accrual unit will increase the priority of point accrual. If the customer shows low purchase intent, the point accrual unit will maintain the normal priority of point accrual. If the customer shows interest in a particular product, the point accrual unit can prioritize point accrual related to that product. This enables rapid point accrual by determining the priority of point accrual based on the customer's purchase intent. Some or all of the above processing in the point accrual unit may be performed using AI or not. For example, the point accrual unit can input customer purchase intent data into an AI model and have the AI ​​model execute the point accrual priority.

[0044] The point addition unit can optimize the points awarded by taking into account the customer's geographical location. For example, if a customer is in a specific region, the unit will prioritize awarding points related to that region. If a customer is on the move, the unit can award points in real time based on their current location. If a customer is in a specific store, the unit can prioritize awarding points related to that store. This allows the unit to provide highly relevant points by optimizing the points awarded based on the customer's geographical location. Some or all of the above processing in the point addition unit may be performed using AI or not. For example, the point addition unit can input the customer's geographical location data into an AI model and have the AI ​​model perform the optimization of the points awarded.

[0045] The addition unit can analyze a customer's social media activity and provide relevant bonus points. For example, if a customer mentions a specific product on social media, the addition unit can add points related to that product. If a customer mentions a specific store on social media, the addition unit can add points related to that store. If a customer mentions a specific event on social media, the addition unit can add points related to that event. This allows for the addition of points tailored to the customer's interests by providing relevant bonus points based on their social media activity. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant bonus points.

[0046] The support department can analyze a customer's past self-checkout usage history and select the optimal support method. For example, if a customer frequently uses self-checkout, the support department can provide detailed support. If a customer is using self-checkout for the first time, the support department can provide basic support. If a customer uses self-checkout during a specific time period, the support department can select the optimal support method for that time period. This improves the efficiency of support by selecting the optimal support method based on the customer's past self-checkout usage history. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer self-checkout usage history data into a generative AI model and have the generative AI model select the optimal support method.

[0047] The support department can determine the priority of support based on the customer's current purchase intent. For example, if the customer shows high purchase intent, the support department will increase the priority of support. If the customer shows low purchase intent, the support department will maintain the normal priority of support. If the customer has shown interest in a particular product, the support department can prioritize support related to that product. This enables prompt support by determining the priority of support based on the customer's purchase intent. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer purchase intent data into a generative AI model and have the generative AI model execute the support prioritization.

[0048] The support department can optimize support content by taking into account the customer's geographical location. For example, if the customer is in a specific region, the support department can provide support content relevant to that region. If the customer is on the move, the support department can provide support content based on their current location in real time. If the customer is in a specific store, the support department can provide support content relevant to that store. In this way, by optimizing support content based on the customer's geographical location, highly relevant support can be provided. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input the customer's geographical location data into a generative AI model and have the generative AI model perform the optimization of support content.

[0049] The support department can analyze customers' social media activity and provide relevant support. For example, if a customer mentions a specific product on social media, the support department can provide support related to that product. If a customer mentions a specific store on social media, the support department can provide support related to that store. If a customer mentions a specific event on social media, the support department can provide support related to that event. This allows the support department to provide support tailored to the customer's interests by providing relevant support based on the customer's social media activity. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer social media activity data into a generative AI model and have the generative AI model perform the provision of relevant support.

[0050] The analysis unit can analyze customers' past self-checkout usage data and select the optimal analysis method. For example, if a customer uses self-checkout frequently, the analysis unit can perform a detailed analysis. If a customer uses self-checkout for the first time, the analysis unit can perform a basic analysis. If a customer uses self-checkout during a specific time period, the analysis unit can select the optimal analysis method for that time period. This improves the efficiency of the analysis by selecting the optimal analysis method based on the customer's past self-checkout usage data. Some or all of the above processes in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer self-checkout usage data into an AI model and have the AI ​​model select the optimal analysis method.

[0051] The analysis department can determine the priority of analyses based on the customer's current purchase intent. For example, if a customer shows high purchase intent, the analysis department will increase its priority. If a customer shows low purchase intent, the analysis department will maintain its normal priority. If a customer shows interest in a particular product, the analysis department can prioritize analyses related to that product. This allows for rapid analysis by determining the priority of analyses based on the customer's purchase intent. Some or all of the above processes in the analysis department may be performed using AI or not. For example, the analysis department can input customer purchase intent data into an AI model and have the AI ​​model perform the analysis prioritization.

[0052] The analysis unit can optimize analysis results by taking into account the customer's geographical location. For example, if a customer is in a specific region, the analysis unit can prioritize displaying information relevant to that region. If a customer is on the move, the analysis unit can display information based on their current location in real time. If a customer is in a specific store, the analysis unit can prioritize displaying information relevant to that store. This allows the analysis unit to provide highly relevant information by optimizing the analysis results based on the customer's geographical location. Some or all of the above processing in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer geographical location data into an AI model and have the AI ​​model perform the optimization of the analysis results.

[0053] The analytics department can analyze customers' social media activity and provide relevant analytical results. For example, if a customer mentions a specific product on social media, the analytics department can display information related to that product. If a customer mentions a specific store on social media, the analytics department can display information related to that store. If a customer mentions a specific event on social media, the analytics department can display information related to that event. This allows the analytics department to display information tailored to the customer's interests by providing relevant analytical results based on their social media activity. Some or all of the above processing in the analytics department may be performed using AI or not. For example, the analytics department can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant analytical results.

[0054] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0055] In addition to facial recognition, the identification unit can recognize the customer's voice and estimate the customer's intent based on the voice recognition results. For example, if a customer says, "I want to use my points," the identification unit recognizes the voice and estimates the customer's intent to use points. Furthermore, if a customer says, "I want to pay by card," the identification unit recognizes the voice and estimates the customer's intent to pay by card. This enables smoother self-checkout operation based on customer voice recognition. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer voice data into an AI model and have the AI ​​model perform intent estimation.

[0056] The point accumulation unit can analyze a customer's purchase history and determine the priority for accumulating points for specific products. For example, it can prioritize point accumulation for products that a customer has frequently purchased in the past. Furthermore, it can prioritize point accumulation for products that a customer purchased during a specific campaign period. This allows for increased customer purchasing intent by determining point accumulation priorities based on the customer's purchase history. Some or all of the above processing in the point accumulation unit may be performed using AI or not. For example, the point accumulation unit can input customer purchase history data into an AI model and have the AI ​​model execute the point accumulation priority.

[0057] The analysis department can analyze customer purchase history and predict purchasing trends for specific products. For example, it can predict purchasing trends for products that customers have frequently purchased in the past. Furthermore, it can predict purchasing trends for products that customers have purchased during a specific campaign period. This allows for the optimization of store inventory management by predicting purchasing trends based on customer purchase history. Some or all of the above processes in the analysis department may be performed using AI or not. For example, the analysis department can input customer purchase history data into an AI model and have the AI ​​model perform purchasing trend predictions.

[0058] In addition to facial recognition, the identification unit can recognize the customer's walking pattern and estimate the customer's intention based on that pattern. For example, if a customer is walking quickly, the identification unit recognizes that walking pattern and estimates the intention to be in a hurry. Furthermore, if a customer is walking slowly, the identification unit recognizes that walking pattern and estimates the intention to be relaxed. This enables smoother self-checkout operation based on the customer's walking pattern. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer walking pattern data into an AI model and have the AI ​​model perform intention estimation.

[0059] The support department can analyze a customer's past self-checkout usage history and select the most appropriate support method. For example, if a customer frequently uses self-checkout, detailed support can be provided. If a customer is using self-checkout for the first time, basic support can be provided. If a customer uses self-checkout during a specific time period, the support department can select the most appropriate support method for that time period. This improves the efficiency of support by selecting the most appropriate support method based on the customer's past self-checkout usage history. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer self-checkout usage history data into a generative AI model and have the generative AI model select the most appropriate support method.

[0060] In addition to facial recognition, the identification unit can measure the customer's body temperature and estimate their health status based on that temperature. For example, if the customer's body temperature is high, the identification unit measures the temperature and estimates that their health status is poor. Furthermore, if the customer's body temperature is low, the identification unit measures the temperature and estimates that their health status is good. This allows for more appropriate self-checkout operations based on the customer's body temperature. Some or all of the above-described processes in the identification unit may be performed using AI or not. For example, the identification unit can input customer body temperature data into an AI model and have the AI ​​model perform the health status estimation.

[0061] The addition unit can analyze a customer's social media activity and provide relevant bonus points. For example, if a customer mentions a specific product on social media, it can add points related to that product. If a customer mentions a specific store on social media, it can add points related to that store. If a customer mentions a specific event on social media, it can add points related to that event. This allows for the addition of points tailored to the customer's interests by providing relevant bonus points based on their social media activity. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant bonus points.

[0062] The following briefly describes the processing flow for example form 1.

[0063] Step 1: The identification unit recognizes the customer's face. The identification unit can, for example, use a facial recognition algorithm to identify the customer's face and use a camera as the sensor. Step 2: The addition unit adds points based on the customer information recognized by the identification unit. The addition unit, for example, sets the unit of points and determines the timing of the addition. The addition unit may include AI processing. Step 3: The support department assists with the operation of the self-checkout machines. The support department uses generative AI to provide real-time support if customers have difficulty scanning items or processing payments. For example, the support department provides voice guidance and on-screen display instructions. Step 4: The analysis department analyzes the operation data from the self-checkout systems. The analysis department then makes suggestions to improve the operational efficiency of the stores, for example, by using data collection methods and analysis algorithms.

[0064] (Example of form 2) The self-checkout system according to an embodiment of the present invention is a new self-checkout system that utilizes facial recognition technology and generative AI to address the labor shortage in Japan's service industry, amidst the spread of self-checkout systems during the COVID-19 pandemic. This self-checkout system recognizes the face of a customer entering the store, and if the customer is registered as a member, it can use facial recognition to add points and make payments without a card. Generative AI supports the operation of the self-checkout system, enabling user-friendly operation. This promotes efficient checkout operations, compensates for the labor shortage in stores, and increases labor productivity per person. As a result, the self-checkout system can significantly improve customer convenience. For example, a system that recognizes the face of a customer entering the store is activated, and facial recognition technology is used to identify the customer's face. If the customer is registered as a member, points can be added and payments can be made without a card. This significantly improves customer convenience. Next, generative AI supports the operation of the self-checkout system. For example, if a customer is confused about scanning products or processing payments, the generative AI provides support in real time, enabling user-friendly operation. This eliminates the disadvantages of self-checkout systems, allowing customers to use them smoothly. Furthermore, the generating AI analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. For example, it can suggest optimal checkout placement to reduce checkout wait times during peak hours and improvements to increase product scanning speed. This improves store operational efficiency and helps compensate for labor shortages. This system will bring about a major transformation in the Japanese retail industry and provide new consumer experiences. For example, customers can enjoy a smooth and efficient shopping experience by using self-checkouts with the support of facial recognition and generating AI. Stores can also increase labor productivity per person by achieving efficient checkout operations and compensating for labor shortages.

[0065] The self-checkout system according to this embodiment comprises an identification unit, an addition unit, a support unit, and an analysis unit. The identification unit recognizes the customer's face. The identification unit identifies the customer's face using, for example, a face recognition algorithm. The identification unit can use, for example, a camera as the sensor. The addition unit adds points based on the customer information recognized by the identification unit. The addition unit sets the unit of points and determines the timing of the addition. The addition unit may include AI processing. The support unit supports the operation of the self-checkout system. The support unit uses generative AI to provide real-time support when the customer is confused about scanning products or processing payments. The support unit provides, for example, voice guidance or on-screen guidance. The analysis unit analyzes the operation data of the self-checkout system. The analysis unit makes suggestions to improve the operational efficiency of the store using, for example, data collection methods and analysis algorithms. As a result, the self-checkout system according to this embodiment is capable of customer face recognition, point addition, self-checkout operation support, and operation data analysis.

[0066] The identification unit recognizes the customer's face. For example, the identification unit identifies the customer's face using a facial recognition algorithm. Specifically, the identification unit uses a high-resolution camera to acquire a facial image of the customer and inputs it into the facial recognition algorithm. The facial recognition algorithm uses a model pre-trained with deep learning technology to extract the customer's facial features. These features include information such as the position of the eyes, the shape of the nose, and the contour of the mouth. Based on these features, the identification unit matches the customer's face against existing facial data in the database to identify the customer. If the match is successful, the identification unit obtains the customer's ID information and passes it on to the next process. To improve the matching accuracy, the identification unit can also perform correction processing that takes into account environmental factors such as lighting conditions and camera angle. Furthermore, the identification unit can achieve more accurate facial recognition by using multiple cameras to acquire facial images from different angles and generating a 3D model. This allows the identification unit to quickly and accurately recognize customer faces and provide efficient service in conjunction with other functions of the self-checkout system.

[0067] The point accrual unit adds points based on customer information recognized by the identification unit. Specifically, the accrual unit receives customer ID information and accesses the customer's points account. The accrual unit sets the unit of points and calculates points based on, for example, the purchase amount. A system can also be implemented where bonus points are added when the purchase amount exceeds a certain threshold. The accrual unit can also use AI to analyze the customer's purchase history and offer additional points for specific products or campaigns. For example, customers who purchase specific products during a specific period can be awarded bonus points in addition to regular points. The accrual unit processes point accrual in real time, ensuring that points are reflected immediately when the customer passes through the checkout. Furthermore, the accrual unit records the point accrual history so that customers can review it later. This allows the accrual unit to provide a transparent points service to customers and improve customer satisfaction.

[0068] The support department assists with the operation of self-checkout machines. Specifically, the support department uses generative AI to provide real-time support when customers encounter difficulties scanning items or processing payments. The generative AI uses natural language processing technology to understand customer questions and problems and generates appropriate answers and instructions. For example, if a customer is unable to scan an item's barcode, the support department will provide voice guidance such as, "Please bring the barcode closer to the scanner." It can also visually display scanning procedures and points to note through on-screen displays. The support department monitors customer operation history in real time and intervenes immediately to provide support when problems occur. Furthermore, the support department can provide personalized support based on the customer's past operation and purchase history. For example, if a particular customer has repeatedly encountered the same problem in the past, special attention can be paid to that customer to prevent the problem from recurring. In this way, the support department can enable customers to use self-checkout machines smoothly and improve the customer experience.

[0069] The analytics department analyzes operation data from self-checkout registers. Specifically, the analytics department collects customer operation and purchase data and uses this data to propose ways to improve store operational efficiency. The analytics department uses self-checkout operation logs, customer purchase history, and point accumulation history as data collection methods. This data is integrated and analyzed using AI. For example, it identifies which operations customers spend the most time on at self-checkout registers and analyzes the causes. It can extract specific problems, such as difficulty scanning certain products or delays in payment processing. The analytics department proposes solutions to these problems, improving store operational efficiency. For example, for products that are difficult to scan, measures such as changing the barcode position can be taken. The analytics department can also analyze customer purchasing patterns and predict congestion levels at specific times of day and on specific days of the week. This allows stores to appropriately allocate staff and manage inventory. Furthermore, the analytics department collects customer feedback and provides data to continuously improve the usability and convenience of self-checkout registers. In this way, the analytics department can not only improve store operational efficiency but also contribute to increased customer satisfaction.

[0070] The support unit can assist customers in real time using generative AI. For example, the support unit can use generative AI to provide real-time support when customers are confused about scanning products or processing payments. The support unit can also use generative AI to provide voice guidance and on-screen instructions. This makes self-checkout operations smoother by assisting customers in real time using generative AI. Some or all of the above processes in the support unit may be performed using generative AI or not. For example, the support unit can use an AI model that uses generative AI to assist customers in real time and support customer operations.

[0071] The analysis unit may include a proposal unit that analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. For example, the analysis unit makes suggestions to improve store operational efficiency using data collection methods and analysis algorithms. The proposal unit can, for example, suggest optimal checkout layouts to reduce checkout waiting times during peak hours, or suggest improvements to increase product scanning speed. This enables suggestions to improve store operational efficiency. Some or all of the above processing in the analysis unit may be performed using AI, or not. For example, the analysis unit can input self-checkout operation data into an AI model and have the AI ​​model execute suggestions to improve store operational efficiency.

[0072] The identification unit recognizes the customer's face and, if the customer is registered, can add points or process payments without a card. The identification unit identifies the customer's face using, for example, a facial recognition algorithm. The identification unit can use, for example, a camera as a sensor. The identification unit can add points or process payments without a card if the customer is registered. This allows for the addition of points and payment without a card based on the customer's facial recognition. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input the customer's facial recognition data into an AI model and have the AI ​​model perform the addition of points or payment.

[0073] The addition unit can add points based on customer facial recognition. The addition unit authenticates the customer's face using, for example, a facial recognition algorithm. The addition unit can use, for example, a camera as the sensor. The addition unit adds points based on customer facial recognition. This allows points to be added based on customer facial recognition. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer facial recognition data into an AI model and have the AI ​​model perform the point addition.

[0074] The identification unit can estimate the customer's emotions and adjust the accuracy of face recognition based on the estimated emotions. For example, if the customer is nervous, the identification unit can increase the accuracy of face recognition to prevent misrecognition. If the customer is relaxed, the identification unit can maintain normal accuracy for smooth recognition. If the customer is in a hurry, the identification unit can prioritize the speed of face recognition for rapid recognition. This prevents misrecognition by adjusting the accuracy of face recognition based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer emotion data into an AI model and have the AI ​​model perform the adjustment of face recognition accuracy.

[0075] The identification unit can analyze a customer's past visit history and select the optimal recognition method. For example, if a customer visits frequently, the identification unit can improve the accuracy of facial recognition to recognize them quickly. If a customer is visiting for the first time, the identification unit can maintain normal facial recognition accuracy and perform detailed recognition. If a customer visits during a specific time period, the identification unit can select the optimal recognition method for that time period. This improves recognition accuracy by selecting the optimal recognition method based on the customer's past visit history. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer visit history data into an AI model and have the AI ​​model select the optimal recognition method.

[0076] The identification unit can determine the recognition priority based on the customer's current purchase intent when recognizing a customer's face. For example, if the customer shows high purchase intent, the identification unit will increase the priority of face recognition to recognize them quickly. If the customer shows low purchase intent, the identification unit will maintain the normal priority of face recognition. If the customer shows interest in a specific product, the identification unit can prioritize recognition related to that product. This enables rapid recognition by determining the recognition priority based on the customer's purchase intent. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer purchase intent data into an AI model and have the AI ​​model execute the recognition priority.

[0077] The identification unit can estimate the customer's emotions and adjust the display method of the recognition results based on the estimated emotions. For example, if the customer is nervous, the identification unit can provide a simple and highly visible display method. If the customer is relaxed, the identification unit can provide a display method that includes detailed information. If the customer is in a hurry, the identification unit can provide a concise display method. This improves visibility by adjusting the display method of the recognition results based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer emotion data into an AI model and have the AI ​​model execute the display method of the recognition results.

[0078] The identification unit can optimize the recognition results by taking into account the customer's geographical location information. For example, if the customer is in a specific region, the identification unit can prioritize displaying information related to that region. If the customer is on the move, the identification unit can display information based on their current location in real time. If the customer is in a specific store, the identification unit can prioritize displaying information related to that store. In this way, by optimizing the recognition results based on the customer's geographical location information, highly relevant information can be provided. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input the customer's geographical location data into an AI model and have the AI ​​model perform the optimization of the recognition results.

[0079] The identification unit can analyze a customer's social media activity and provide relevant recognition results. For example, if a customer mentions a specific product on social media, the identification unit can display information related to that product. If a customer mentions a specific store on social media, the identification unit can display information related to that store. If a customer mentions a specific event on social media, the identification unit can display information related to that event. In this way, by providing relevant recognition results based on the customer's social media activity, information tailored to the customer's interests can be displayed. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer social media activity data into an AI model and have the AI ​​model perform the task of providing relevant recognition results.

[0080] The addition unit can estimate the customer's emotions and adjust the timing of point addition based on the estimated emotions. For example, if the customer is relaxed, the addition unit can add points at the normal timing. If the customer is in a hurry, the addition unit can add points quickly. If the customer is excited, the addition unit can adjust the timing of point addition to add points at an appropriate time. This allows points to be added at an appropriate time by adjusting the timing of point addition based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the addition unit may be performed using AI or not using AI. For example, the addition unit can input customer emotion data into an AI model and have the AI ​​model execute the timing of point addition.

[0081] The point accrual unit can analyze a customer's past point usage history and select the optimal accrual method. For example, if a customer uses points frequently, the accrual unit can increase the frequency of point accrual. If a customer uses points at a specific time, the accrual unit can accrue points at that time. If a customer uses points for a specific product, the accrual unit can select a point accrual method related to that product. This improves the efficiency of point accrual by selecting the optimal accrual method based on the customer's past point usage history. Some or all of the above processing in the accrual unit may be performed using AI or not. For example, the accrual unit can input customer point usage history data into an AI model and have the AI ​​model select the optimal accrual method.

[0082] The point accrual unit can determine the priority of point accrual based on the customer's current purchase intent. For example, if the customer shows high purchase intent, the point accrual unit will increase the priority of point accrual. If the customer shows low purchase intent, the point accrual unit will maintain the normal priority of point accrual. If the customer shows interest in a particular product, the point accrual unit can prioritize point accrual related to that product. This enables rapid point accrual by determining the priority of point accrual based on the customer's purchase intent. Some or all of the above processing in the point accrual unit may be performed using AI or not. For example, the point accrual unit can input customer purchase intent data into an AI model and have the AI ​​model execute the point accrual priority.

[0083] The addition unit can estimate the customer's emotions and adjust the display method of the added points based on the estimated customer emotions. For example, if the customer is nervous, the addition unit can provide a simple and highly visible display method. If the customer is relaxed, the addition unit can provide a display method that includes detailed information. If the customer is in a hurry, the addition unit can provide a concise display method. This improves visibility by adjusting the display method of the added points based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the addition unit may be performed using AI or not using AI. For example, the addition unit can input customer emotion data into an AI model and have the AI ​​model execute the display method of the added points.

[0084] The point addition unit can optimize the points awarded by taking into account the customer's geographical location. For example, if a customer is in a specific region, the unit will prioritize awarding points related to that region. If a customer is on the move, the unit can award points in real time based on their current location. If a customer is in a specific store, the unit can prioritize awarding points related to that store. This allows the unit to provide highly relevant points by optimizing the points awarded based on the customer's geographical location. Some or all of the above processing in the point addition unit may be performed using AI or not. For example, the point addition unit can input the customer's geographical location data into an AI model and have the AI ​​model perform the optimization of the points awarded.

[0085] The addition unit can analyze a customer's social media activity and provide relevant bonus points. For example, if a customer mentions a specific product on social media, the addition unit can add points related to that product. If a customer mentions a specific store on social media, the addition unit can add points related to that store. If a customer mentions a specific event on social media, the addition unit can add points related to that event. This allows for the addition of points tailored to the customer's interests by providing relevant bonus points based on their social media activity. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant bonus points.

[0086] The support unit can estimate the customer's emotions and adjust the support content based on those emotions. For example, if the customer is nervous, the support unit can provide simple and easy-to-understand support. If the customer is relaxed, the support unit can provide support that includes detailed information. If the customer is in a hurry, the support unit can provide concise support. This improves readability by adjusting the support content based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the support unit may be performed using generative AI or not. For example, the support unit can input customer emotion data into a generative AI model and have the generative AI model adjust the support content.

[0087] The support department can analyze a customer's past self-checkout usage history and select the optimal support method. For example, if a customer frequently uses self-checkout, the support department can provide detailed support. If a customer is using self-checkout for the first time, the support department can provide basic support. If a customer uses self-checkout during a specific time period, the support department can select the optimal support method for that time period. This improves the efficiency of support by selecting the optimal support method based on the customer's past self-checkout usage history. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer self-checkout usage history data into a generative AI model and have the generative AI model select the optimal support method.

[0088] The support department can determine the priority of support based on the customer's current purchase intent. For example, if the customer shows high purchase intent, the support department will increase the priority of support. If the customer shows low purchase intent, the support department will maintain the normal priority of support. If the customer has shown interest in a particular product, the support department can prioritize support related to that product. This enables prompt support by determining the priority of support based on the customer's purchase intent. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer purchase intent data into a generative AI model and have the generative AI model execute the support prioritization.

[0089] The support unit can estimate the customer's emotions and adjust the way support is displayed based on those emotions. For example, if the customer is nervous, the support unit can provide a simple and highly visible display. If the customer is relaxed, the support unit can provide a display that includes detailed information. If the customer is in a hurry, the support unit can provide a concise display. This improves visibility by adjusting the support display based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the support unit may be performed using generative AI or not. For example, the support unit can input customer emotion data into a generative AI model and have the generative AI model execute the support display method.

[0090] The support department can optimize support content by taking into account the customer's geographical location. For example, if the customer is in a specific region, the support department can provide support content relevant to that region. If the customer is on the move, the support department can provide support content based on their current location in real time. If the customer is in a specific store, the support department can provide support content relevant to that store. In this way, by optimizing support content based on the customer's geographical location, highly relevant support can be provided. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input the customer's geographical location data into a generative AI model and have the generative AI model perform the optimization of support content.

[0091] The support department can analyze customers' social media activity and provide relevant support. For example, if a customer mentions a specific product on social media, the support department can provide support related to that product. If a customer mentions a specific store on social media, the support department can provide support related to that store. If a customer mentions a specific event on social media, the support department can provide support related to that event. This allows the support department to provide support tailored to the customer's interests by providing relevant support based on the customer's social media activity. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer social media activity data into a generative AI model and have the generative AI model perform the provision of relevant support.

[0092] The analysis unit can estimate the customer's emotions and adjust the accuracy of the analysis based on the estimated emotions. For example, if the customer is relaxed, the analysis unit will perform the analysis with normal accuracy. If the customer is stressed, the analysis unit can increase the accuracy of the analysis to prevent misrecognition. If the customer is in a hurry, the analysis unit can prioritize the speed of the analysis and provide results quickly. This prevents misrecognition by adjusting the accuracy of the analysis based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer emotion data into an AI model and have the AI ​​model perform the adjustment of the analysis accuracy.

[0093] The analysis unit can analyze customers' past self-checkout usage data and select the optimal analysis method. For example, if a customer uses self-checkout frequently, the analysis unit can perform a detailed analysis. If a customer uses self-checkout for the first time, the analysis unit can perform a basic analysis. If a customer uses self-checkout during a specific time period, the analysis unit can select the optimal analysis method for that time period. This improves the efficiency of the analysis by selecting the optimal analysis method based on the customer's past self-checkout usage data. Some or all of the above processes in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer self-checkout usage data into an AI model and have the AI ​​model select the optimal analysis method.

[0094] The analysis department can determine the priority of analyses based on the customer's current purchase intent. For example, if a customer shows high purchase intent, the analysis department will increase its priority. If a customer shows low purchase intent, the analysis department will maintain its normal priority. If a customer shows interest in a particular product, the analysis department can prioritize analyses related to that product. This allows for rapid analysis by determining the priority of analyses based on the customer's purchase intent. Some or all of the above processes in the analysis department may be performed using AI or not. For example, the analysis department can input customer purchase intent data into an AI model and have the AI ​​model perform the analysis prioritization.

[0095] The analysis unit can estimate the customer's emotions and adjust how the analysis results are displayed based on the estimated emotions. For example, if the customer is nervous, the analysis unit can provide a simple and easy-to-read display. If the customer is relaxed, the analysis unit can provide a display that includes detailed information. If the customer is in a hurry, the analysis unit can provide a concise display. This improves readability by adjusting how the analysis results are displayed based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer emotion data into an AI model and have the AI ​​model execute how the analysis results are displayed.

[0096] The analysis unit can optimize analysis results by taking into account the customer's geographical location. For example, if a customer is in a specific region, the analysis unit can prioritize displaying information relevant to that region. If a customer is on the move, the analysis unit can display information based on their current location in real time. If a customer is in a specific store, the analysis unit can prioritize displaying information relevant to that store. This allows the analysis unit to provide highly relevant information by optimizing the analysis results based on the customer's geographical location. Some or all of the above processing in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer geographical location data into an AI model and have the AI ​​model perform the optimization of the analysis results.

[0097] The analytics department can analyze customers' social media activity and provide relevant analytical results. For example, if a customer mentions a specific product on social media, the analytics department can display information related to that product. If a customer mentions a specific store on social media, the analytics department can display information related to that store. If a customer mentions a specific event on social media, the analytics department can display information related to that event. This allows the analytics department to display information tailored to the customer's interests by providing relevant analytical results based on their social media activity. Some or all of the above processing in the analytics department may be performed using AI or not. For example, the analytics department can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant analytical results.

[0098] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0099] In addition to facial recognition, the identification unit can recognize the customer's voice and estimate the customer's intent based on the voice recognition results. For example, if a customer says, "I want to use my points," the identification unit recognizes the voice and estimates the customer's intent to use points. Furthermore, if a customer says, "I want to pay by card," the identification unit recognizes the voice and estimates the customer's intent to pay by card. This enables smoother self-checkout operation based on customer voice recognition. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer voice data into an AI model and have the AI ​​model perform intent estimation.

[0100] The point accumulation unit can analyze a customer's purchase history and determine the priority for accumulating points for specific products. For example, it can prioritize point accumulation for products that a customer has frequently purchased in the past. Furthermore, it can prioritize point accumulation for products that a customer purchased during a specific campaign period. This allows for increased customer purchasing intent by determining point accumulation priorities based on the customer's purchase history. Some or all of the above processing in the point accumulation unit may be performed using AI or not. For example, the point accumulation unit can input customer purchase history data into an AI model and have the AI ​​model execute the point accumulation priority.

[0101] The support unit can estimate the customer's emotions and adjust the support content based on those emotions. For example, if the customer is nervous, the support unit can provide simple and easy-to-understand support. If the customer is relaxed, it can provide support that includes detailed information. If the customer is in a hurry, it can provide support that gets straight to the point. This improves readability by adjusting the support content based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the support unit may be performed using generative AI or not. For example, the support unit can input customer emotion data into a generative AI model and have the generative AI model adjust the support content.

[0102] The analysis department can analyze customer purchase history and predict purchasing trends for specific products. For example, it can predict purchasing trends for products that customers have frequently purchased in the past. Furthermore, it can predict purchasing trends for products that customers have purchased during a specific campaign period. This allows for the optimization of store inventory management by predicting purchasing trends based on customer purchase history. Some or all of the above processes in the analysis department may be performed using AI or not. For example, the analysis department can input customer purchase history data into an AI model and have the AI ​​model perform purchasing trend predictions.

[0103] In addition to facial recognition, the identification unit can recognize the customer's walking pattern and estimate the customer's intention based on that pattern. For example, if a customer is walking quickly, the identification unit recognizes that walking pattern and estimates the intention to be in a hurry. Furthermore, if a customer is walking slowly, the identification unit recognizes that walking pattern and estimates the intention to be relaxed. This enables smoother self-checkout operation based on the customer's walking pattern. Some or all of the above processing in the identification unit may be performed using AI or not. For example, the identification unit can input customer walking pattern data into an AI model and have the AI ​​model perform intention estimation.

[0104] The addition unit can estimate the customer's emotions and adjust the timing of point addition based on the estimated emotions. For example, if the customer is relaxed, points can be added at the normal timing. If the customer is in a hurry, points can be added quickly. If the customer is excited, the timing of point addition can be adjusted to add points at an appropriate time. In this way, points can be added at an appropriate time by adjusting the timing of point addition based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the addition unit may be performed using AI or not using AI. For example, the addition unit can input customer emotion data into an AI model and have the AI ​​model execute the timing of point addition.

[0105] The support department can analyze a customer's past self-checkout usage history and select the most appropriate support method. For example, if a customer frequently uses self-checkout, detailed support can be provided. If a customer is using self-checkout for the first time, basic support can be provided. If a customer uses self-checkout during a specific time period, the support department can select the most appropriate support method for that time period. This improves the efficiency of support by selecting the most appropriate support method based on the customer's past self-checkout usage history. Some or all of the above processing in the support department may be performed using generative AI, or not. For example, the support department can input customer self-checkout usage history data into a generative AI model and have the generative AI model select the most appropriate support method.

[0106] The analysis unit can estimate the customer's emotions and adjust the accuracy of the analysis based on the estimated emotions. For example, if the customer is relaxed, the analysis is performed with normal accuracy. If the customer is stressed, the accuracy of the analysis can be increased to prevent misrecognition. If the customer is in a hurry, the speed of the analysis can be prioritized to provide results quickly. This prevents misrecognition by adjusting the accuracy of the analysis based on the customer's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using AI or not. For example, the analysis unit can input customer emotion data into an AI model and have the AI ​​model perform the adjustment of the analysis accuracy.

[0107] In addition to facial recognition, the identification unit can measure the customer's body temperature and estimate their health status based on that temperature. For example, if the customer's body temperature is high, the identification unit measures the temperature and estimates that their health status is poor. Furthermore, if the customer's body temperature is low, the identification unit measures the temperature and estimates that their health status is good. This allows for more appropriate self-checkout operations based on the customer's body temperature. Some or all of the above-described processes in the identification unit may be performed using AI or not. For example, the identification unit can input customer body temperature data into an AI model and have the AI ​​model perform the health status estimation.

[0108] The addition unit can analyze a customer's social media activity and provide relevant bonus points. For example, if a customer mentions a specific product on social media, it can add points related to that product. If a customer mentions a specific store on social media, it can add points related to that store. If a customer mentions a specific event on social media, it can add points related to that event. This allows for the addition of points tailored to the customer's interests by providing relevant bonus points based on their social media activity. Some or all of the above processing in the addition unit may be performed using AI or not. For example, the addition unit can input customer social media activity data into an AI model and have the AI ​​model perform the provision of relevant bonus points.

[0109] The following briefly describes the processing flow for example form 2.

[0110] Step 1: The identification unit recognizes the customer's face. The identification unit can, for example, use a facial recognition algorithm to identify the customer's face and use a camera as the sensor. Step 2: The addition unit adds points based on the customer information recognized by the identification unit. The addition unit, for example, sets the unit of points and determines the timing of the addition. The addition unit may include AI processing. Step 3: The support department assists with the operation of the self-checkout machines. The support department uses generative AI to provide real-time support if customers have difficulty scanning items or processing payments. For example, the support department provides voice guidance and on-screen display instructions. Step 4: The analysis department analyzes the operation data from the self-checkout systems. The analysis department then makes suggestions to improve the operational efficiency of the stores, for example, by using data collection methods and analysis algorithms.

[0111] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0112] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0113] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0114] Each of the multiple elements described above, including the identification unit, addition unit, support unit, and analysis unit, is implemented, for example, in at least one of the smart device 14 and the data processing unit 12. For example, the identification unit uses the camera 42 of the smart device 14 to recognize the customer's face and the control unit 46A executes a face recognition algorithm. The addition unit performs point addition processing, for example, using the identification processing unit 290 of the data processing unit 12. The support unit provides real-time support using generated AI, for example, using the control unit 46A of the smart device 14. The analysis unit analyzes self-checkout operation data, for example, using the identification processing unit 290 of the data processing unit 12 and makes suggestions to improve the operational efficiency of the store. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0115] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0116] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0117] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0118] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0119] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0120] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0121] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0122] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0123] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0124] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0125] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0126] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0127] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0128] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0129] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0130] Each of the multiple elements described above, including the identification unit, addition unit, support unit, and analysis unit, is implemented, for example, in at least one of the smart glasses 214 and the data processing unit 12. For example, the identification unit uses the camera 42 of the smart glasses 214 to recognize the customer's face and the control unit 46A executes a face recognition algorithm. The addition unit performs point addition processing, for example, using the identification processing unit 290 of the data processing unit 12. The support unit provides real-time support using generated AI, for example, using the control unit 46A of the smart glasses 214. The analysis unit analyzes self-checkout operation data, for example, using the identification processing unit 290 of the data processing unit 12 and makes suggestions to improve the operational efficiency of the store. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0131] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0132] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0133] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0134] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0135] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0136] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0137] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0138] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0139] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0140] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0141] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0142] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0143] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0144] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0145] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0146] Each of the multiple elements described above, including the identification unit, addition unit, support unit, and analysis unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the identification unit uses the camera 42 of the headset terminal 314 to recognize the customer's face and the control unit 46A executes a face recognition algorithm. The addition unit performs point addition processing using the identification processing unit 290 of the data processing unit 12. The support unit provides real-time support using generated AI, using the control unit 46A of the headset terminal 314. The analysis unit analyzes self-checkout operation data using the identification processing unit 290 of the data processing unit 12 and makes suggestions to improve the operational efficiency of the store. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0147] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0148] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0149] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0150] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0151] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0152] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0153] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0154] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0155] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0156] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0157] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0158] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0159] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0160] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0161] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0162] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0163] Each of the multiple elements described above, including the identification unit, addition unit, support unit, and analysis unit, is implemented in, for example, at least one of the robot 414 and the data processing unit 12. For example, the identification unit uses the camera 42 of the robot 414 to recognize a customer's face and the control unit 46A executes a face recognition algorithm. The addition unit performs point addition processing using, for example, the identification processing unit 290 of the data processing unit 12. The support unit provides real-time support using generated AI, for example, the control unit 46A of the robot 414. The analysis unit analyzes self-checkout operation data using, for example, the identification processing unit 290 of the data processing unit 12 and makes suggestions to improve the operational efficiency of the store. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0164] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0165] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0166] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0167] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0168] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0169] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0170] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0171] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0172] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0173] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0174] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0175] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0176] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0177] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0178] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0179] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0180] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0181] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0182] (Note 1) An identification unit that recognizes the customer's face, An adding unit that adds points based on the customer information recognized by the identification unit, A support unit that assists with the operation of the self-checkout system, It includes an analysis unit that analyzes the operation data of the self-checkout system. A system characterized by the following features. (Note 2) The aforementioned support unit is Generative AI provides real-time customer support. The system described in Appendix 1, characterized by the features described herein. (Note 3) The aforementioned analysis unit is The company has a proposal department that analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned identification unit is The system recognizes the customer's face and, if they are a registered member, allows for point accumulation and payment without the need for a card. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned addition unit is Points are awarded based on customer facial recognition. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned identification unit is It estimates the customer's emotions and adjusts the accuracy of facial recognition based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned identification unit is Analyze the customer's past visit history and select the optimal recognition method. The system described in Appendix 1, characterized by the features described herein. (Note 8) The aforementioned identification unit is When recognizing a customer's face, the system prioritizes recognition based on the customer's current purchasing intent. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned identification unit is It estimates customer emotions and adjusts how the recognition results are displayed based on the estimated customer emotions. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned identification unit is Optimize recognition results by taking into account the customer's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned identification unit is We analyze customers' social media activity and provide relevant insights. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned addition unit is We estimate customer emotions and adjust the timing of point accrual based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned addition unit is We analyze the customer's past point usage history and select the optimal method for accumulating points. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned addition unit is Prioritizing point accrual based on the customer's current purchasing intent. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned addition unit is The system estimates customer emotions and adjusts how points are displayed based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned addition unit is Optimize bonus points by taking into account the customer's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned addition unit is Analyze customer social media activity and provide relevant bonus points. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned support unit is We estimate the customer's emotions and adjust the support provided based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned support unit is We analyze the customer's past self-checkout usage history to select the most suitable support method. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned support unit is Prioritize support based on the customer's current purchasing intent. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned support unit is We estimate customer sentiment and adjust how support is displayed based on that estimated sentiment. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned support unit is We optimize support services by taking into account the customer's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned support unit is We analyze customers' social media activity and provide relevant support. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned analysis unit is We estimate customer emotions and adjust the accuracy of the analysis based on the estimated customer emotions. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned analysis unit is We analyze customers' past self-checkout usage data and select the most suitable analysis method. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned analysis unit is Prioritize analysis based on the customer's current purchasing intent. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned analysis unit is It estimates customer emotions and adjusts how the analysis results are displayed based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned analysis unit is Optimize analysis results by taking into account the customer's geographical location. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned analysis unit is We analyze customers' social media activity and provide relevant analytical results. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0183] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. An identification unit that recognizes the customer's face, An adding unit that adds points based on the customer information recognized by the identification unit, A support unit that assists with the operation of the self-checkout system, It includes an analysis unit that analyzes the operation data of the self-checkout system. A system characterized by the following features.

2. The aforementioned support unit is Generative AI provides real-time support to customers. The system according to feature 1.

3. The aforementioned analysis unit is The company has a proposal department that analyzes self-checkout operation data and makes suggestions to improve store operational efficiency. The system according to feature 1.

4. The aforementioned identification unit is The system recognizes the customer's face and, if they are a registered member, allows for point accumulation and payment without the need for a card. The system according to feature 1.

5. The aforementioned addition unit is Points are awarded based on customer facial recognition. The system according to feature 1.

6. The aforementioned identification unit is It estimates the customer's emotions and adjusts the accuracy of facial recognition based on the estimated emotions. The system according to feature 1.

7. The aforementioned identification unit is Analyze the customer's past visit history and select the optimal recognition method. The system according to feature 1.

8. The aforementioned identification unit is When recognizing a customer's face, the system prioritizes recognition based on the customer's current purchasing intent. The system according to feature 1.

9. The aforementioned identification unit is It estimates customer emotions and adjusts how the recognition results are displayed based on the estimated customer emotions. The system according to feature 1.

10. The aforementioned identification unit is Optimize recognition results by taking into account the customer's geographical location. The system according to feature 1.

Images