Information processing method, commodity presentation method, information processing device, commodity presentation device, and program

By associating product information with sound production parameters, the method effectively conveys sensory experiences like taste, smell, and touch through information terminals, enhancing user experience by enabling vivid auditory representations of product qualities.

JP2024166832AInactive Publication Date: 2024-11-29YAMAHA CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023083208
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing information processing methods fail to convey human sensations such as taste, smell, and touch effectively through information terminals.

Method used

An information processing method that associates product information with sound production parameters to convey sensory information like taste, smell, or touch by preparing a converter that defines correspondences between product information and sound creation parameters, using trained models to generate sound information based on sensory input or parameters.

Benefits of technology

Enables the intuitive conveyance of human sensations beyond vision by correlating visual and sound information, allowing users to imagine product tastes more vividly through synthesized sounds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166832000001_ABST
    Figure 2024166832000001_ABST
Patent Text Reader

Abstract

To provide an information processing method capable of transmitting human sensation other than visual sensation which is hard to transmit with an information terminal.SOLUTION: An information processing method prepares a converter which defines correspondence between commodity information containing sensation information regarding taste sensation, smell sensation, or tactile sensation among information regarding commodities and parameters regarding sound production, and outputs sound information based on a parameter regarding sound production by inputting the commodity information to the converter, or outputs the commodity information by inputting the parameter regarding the sound production based on the acquired sound information to the converter.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing method, a product presentation method, an information processing device, a product presentation device, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for predicting the quality of food and beverages using deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-18354 A Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned prior art does not convey human senses other than sight, which is difficult to convey through an information terminal.

[0005] One aspect of the present disclosure aims to provide an information processing method that can convey human senses other than vision, which are difficult to convey using an information terminal. [Means for solving the problem]

[0006] An information processing method according to one embodiment of the present invention includes preparing a converter that defines a correspondence between product information, including sensory information relating to taste, smell or touch, and parameters related to sound creation, and inputting the product information into the converter and outputting sound information based on the parameters related to sound creation, or inputting the parameters related to sound creation based on acquired sound information into the converter and outputting the product information.

[0007] In addition, a product information presentation method according to one embodiment of the present invention prepares a converter that defines the correspondence between product information, which includes visual information about a product and sensory information about the product related to taste, smell or touch, and parameters related to sound creation, determines the parameters related to sound creation, and presents the visual information in correspondence with sound information generated based on the parameters related to sound creation. Effect of the Invention

[0008] According to one embodiment of the present invention, human senses other than vision, which are difficult to convey using an information terminal, can be conveyed by using sound information based on sound creation parameters. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing the configuration of a user terminal 12. [Diagram 2] FIG. 2 is an external view of a user terminal 12, showing an example of a display screen (GUI) related to an application program. [Diagram 3] 11 is a flowchart showing the operation of an information processing device at the stage of constructing a trained model. [Figure 4] FIG. 2 is a diagram showing the relationship between component information, taste information, and sound creation parameters. [Diagram 5] 11 is a flowchart illustrating the operation of an information processing device at a stage of using a trained model. [Figure 6] FIG. 13 is an external view of a user terminal 12 showing a display screen (GUI) related to an application program according to a second modified example. [Figure 7] 13 is a flowchart showing an operation in a Synth mode according to Modification 2. [Figure 8] FIG. 2 is an external view of the user terminal 12, showing a GUI screen for receiving sound information. [Figure 9] 13 is an external view of the user terminal 12, showing another example of a GUI screen for receiving sound information. FIG. [Figure 10] FIG. 2 is an external view of a user terminal 12, showing an example of a product information presentation screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Fig. 1 is a block diagram showing the configuration of the user terminal 12. Fig. 1 is an external view of the user terminal 12, showing an example of a display screen related to an application program.

[0011] The user terminal 12 is an information processing device such as a personal computer or a smartphone. The user terminal 12 includes a display 201, a user I / F 202, a flash memory 203, a CPU 204, a RAM 205, a communication I / F 206, a speaker 207, and a microphone 208.

[0012] The display 201 is, for example, an LCD or an OLED, and displays various information. The user I / F 202 is a touch panel that is laminated on the LCD or OLED of the display 201. Alternatively, the user I / F 202 may be a keyboard or a mouse. When the user I / F 202 is a touch panel, the user I / F 202 and the display 201 configure a GUI (Graphical User Interface).

[0013] CPU 204 is an example of a processor, and is a control unit that controls the operation of user terminal 12. CPU 204 performs various operations by reading predetermined programs, such as application programs, stored in flash memory 203, which is a storage medium, into RAM 205 and executing them. The programs may be stored in a server (not shown). CPU 204 may download the programs from the server via a network and execute them.

[0014] FIG. 2 is an external view of the user terminal 12, showing an example of a display screen (GUI) related to an application program. The CPU 204 displays a GUI screen as shown in FIG. 2 on the display 201. The GUI screen is, for example, a screen on a mail-order website. The GUI screen displays a product image 51 and a product name 52 as visual information related to the product. In the example of FIG. 2, the visual information also displays a rating, a capacity, a price, etc. Visual information is information related to a product that can be visually perceived, such as a product name, a package, a logo, a product description, an advertisement, etc. As an example, the product image 51 displays an image of a certain sake, and the product name 52 displays the name of a certain sake, Junmai Ginjo A. The GUI screen also displays a sound icon 53. When the CPU 204 accepts a touch operation on the sound icon 53, it obtains sound creation parameters associated with the product, and reproduces sound information based on the sound creation parameters. The sound information based on the sound creation parameters is different from an instantaneous sound or a voice guidance sound. For example, the CPU 204 obtains parameters (such as attack, decay, and sustain parameters) for synthesizing the sound of a certain sound source, synthesizes the sound of an analog synthesizer based on the parameters, and reproduces the sound of the analog synthesizer. That is, the CPU 204 presents visual information and sound information related to a product on the display 201 in association with each other.

[0015] The CPU 204 acquires sound information corresponding to the product information from the flash memory 203 or from the server via the communication I / F 206, or acquires parameters corresponding to the product information and generates sound information based on the parameters. The flash memory 203 or the server stores sound information corresponding to each of a plurality of products. The sound information is obtained by a converter. The converter is obtained based on a trained model, for example.

[0016] 3 is a flowchart showing the operation of the information processing device at the stage of constructing a trained model. The construction of the trained model may be performed by the CPU 204 of the user terminal 12, or may be performed by a processor of the server. Hereinafter, the operation of constructing the trained model will be referred to as being performed by the processor even when the operation is performed by the CPU 204 of the user terminal 12.

[0017] The processor first receives product information (S11). The product information includes at least a product image or a product name. The processor then obtains ingredient information corresponding to the received product information (S12).

[0018] FIG. 4 is a diagram showing the relationship between component information, taste information, and sound creation parameters. The component information includes at least the component names and component amounts. In the example of FIG. 4, the component names of a certain sake, Junmai Ginjo A, include acetaldehyde, methanol, ethyl acetate, n-propanol, i-amyl alcohol, and i-butyl alcohol, and the respective component amounts. The component information is stored in a database such as flash memory 203 or a server (not shown) as a result of analyzing the product in advance. In other words, the visual information of the product and the component information of the product are associated in advance.

[0019] The processor receives taste information (S13). Taste information is an example of sensory information. As shown in FIG. 4, taste information includes taste names such as sweetness, sourness, saltiness, bitterness, and umami, and evaluation values ​​that indicate the strength of these tastes. Alternatively, taste information may include more detailed evaluation values ​​such as fullness, richness, lightness, or smoothness. Furthermore, taste information may include hierarchical information such as a fullness that spreads slowly in the mouth, a fullness that spreads quickly, and the like.

[0020] Such evaluation values ​​of taste information are defined based on subjective evaluations by many users. The product in this embodiment is sake, and many users (tasters) input their sensations when tasting the sake. The many tasters taste and input taste information. As described above, each piece of product information is associated with component information. Therefore, by the many tasters inputting taste information corresponding to each piece of product information, a data set showing the relationship between the component information and the taste information is prepared. The processor trains a predetermined network model to determine the relationship between the component information and the taste information using a predetermined algorithm and the above data set (S14).

[0021] The algorithm for training the network model is not limited. The processor can train the network model using any algorithm such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), or RNN (Recurrent Neural Network). The processor may also train the network model using HMM (Hidden Markov Model), SVM (Support Vector Machine), or the like.

[0022] This constructs a first trained model F(m)=s that takes the ingredient information m of the product information as input and outputs an evaluation value s of the taste information.

[0023] Next, the processor accepts sound creation parameters (S15). The sound creation parameters are parameters for synthesizing the sound of a certain sound source, and include, for example, parameters such as harmonic structure, pitch, envelope (time parameters such as attack, decay, sustain, and release), and filters, as shown in Fig. 4. Alternatively, the sound creation parameters also include information specifying the sampled sound data (PCM sound source) to be played.

[0024] Such sound creation parameters are input by, for example, a person who specializes in sound creation (hereinafter referred to as a sound creation craftsman). One or more sound creation craftsmen create sound creation parameters for each of the evaluation values ​​of taste information from a large number of users. Alternatively, one or more sound creation craftsmen may each taste sake and create sound creation parameters for each piece of product information. In this way, a data set showing the relationship between taste information and sound creation parameters is prepared.

[0025] The processor trains the relationship between the taste information and the sound creation parameters created by the sound creator using a predetermined algorithm and the above data set in a predetermined network model (S16). This constructs a second trained model G(s)=r that inputs the evaluation value s of the taste information and outputs the sound creation parameters r.

[0026] In this manner, a converter that is made up of the first trained model and the second trained model and converts the product information and the sound creation parameters is prepared. The prepared converter is stored in the flash memory 203 or the server.

[0027] Fig. 5 is a flowchart showing the operation of the information processing device at the stage of using the trained model. When the CPU 204 receives a touch operation of the sound icon 53 on the GUI screen shown in Fig. 2, the CPU 204 starts the operation of the flowchart shown in Fig. 5. First, the CPU 204 receives product information of the product that has been touched (S21). The CPU 204 acquires ingredient information corresponding to the received product information (s22). The product information is, for example, a product name.

[0028] The CPU 204 inputs the product information into a database (such as the flash memory 203 or a server (not shown)) that defines ingredient information corresponding to the product information, and acquires the corresponding ingredient information.

[0029] In addition, the CPU 204 may construct a trained model that receives product information as input and outputs ingredient information, and obtain the ingredient information using the trained model.

[0030] The CPU 204 uses the first trained model to acquire an evaluation value of taste information corresponding to the acquired component information (S23). The CPU 204 also uses the acquired second trained model to acquire sound creation parameters corresponding to the taste information (S24). The CPU 204 outputs sound information based on the acquired sound creation parameters (S25). For example, when the sound creation parameters are parameters for synthesizing the sound of a sound source, the CPU 204 synthesizes a sound signal from the sound source based on the parameters and outputs the synthesized sound signal. When the sound creation parameters are information specifying sound data (PCM sound source) to be reproduced, the CPU 204 reproduces the specified sound data and outputs it as a sound signal. The sound signal output by the CPU 204 is emitted from the speaker 207, for example. That is, in this embodiment, the CPU 204 prepares a converter that specifies the correspondence between product information including sensory information related to taste among information related to the product and parameters related to sound creation, inputs the product information to the converter, and outputs sound information based on the parameters related to sound creation. Furthermore, the CPU 204 prepares a converter that specifies the correspondence between product information, including visual information about the product and sensory information about the taste of the product, and parameters related to sound creation, determines the parameters related to sound creation, and presents the visual information in correspondence with sound information based on the sound creation parameters on a GUI screen such as that shown in FIG. 2.

[0031] As a result, the user hears the sound emitted from the speaker 207. The sound is created by a sound craftsman so that it correlates with sensory information related to taste. Sound has multidimensional elements such as time change and frequency components, and information that changes in the time axis direction is developed by a few seconds of pronunciation, so that more information can be conveyed. Such expression of taste by sound can evoke taste more intuitively than when the strength of sweetness, sourness, etc. is visually expressed as a numerical value. For example, a product with a strong sweetness is emitted as a full, rounded, rich sound with suppressed high frequencies, which is a sound that is highly correlated with the strength of sweetness. Also, a product with a strong smooth taste is expressed as a smooth sound with few odd-order high-order harmonics. Also, for example, a feeling of fullness that slowly spreads in the mouth is expressed as a strong low-frequency sound, and a feeling of fullness that quickly spreads is expressed as a strong high-frequency sound. Therefore, the user can easily imagine the taste of the product by listening to the sound, which is a new customer experience that was not possible before.

[0032] (Variation 1) The processor according to the first modification receives the degree of agreement between the evaluation value and the sound information from the evaluator, and corrects the first trained model based on the received evaluation. The degree of agreement is determined based on the subjective evaluation of the evaluator. The evaluator may be a user of the product (a taster in the above embodiment) or a sound craftsman. The evaluator listens to a sound based on the sound creation parameters obtained by the second trained model G(s)=r, and inputs the degree of agreement ω with the taste information. The processor uses the received degree of agreement ω to minimize the error L between the second trained model G(s) and the output value r, as shown in the following formula (1).

[0033]

number

[0034] In formula (1), ωn is the degree of agreement between the evaluation value s of the nth product and the sound creation parameter r. The second trained model G(s) is further optimized by inputting the degree of agreement ω from multiple evaluators.

[0035] This allows users to have a new customer experience that was not possible before: by listening to the sound, they can more easily imagine the taste of the product.

[0036] (Variation 2) The processor of variant example 2 uses a converter that specifies the correspondence between product information, including sensory information related to taste, smell or touch, and parameters related to sound creation, and inputs parameters related to sound creation based on the acquired sound information to the converter and outputs the product information.

[0037] Fig. 6 is an external view of the user terminal 12, showing a display screen (GUI) related to an application program according to Modification 2. The CPU 204 displays a GUI screen as shown in Fig. 6 on the display 201. The GUI screen displays a listening mode reception icon 71 and a Sake Synth mode reception icon 72.

[0038] When the CPU 204 receives a touch operation on the listening sake mode reception icon 71, it executes the product presentation method of the flowchart shown in Fig. 5. On the other hand, when the CPU 204 receives a touch operation on the sake Synth mode reception icon 72, it executes the process of the flowchart shown in Fig. 7. Fig. 7 is a flowchart showing the operation of the sake Synth mode according to Modification 2. The sake Synth mode is a mode in which, by inputting sound information, information on the corresponding product (sake in this embodiment) is presented.

[0039] First, the CPU 204 accepts sound information (S31). Fig. 8 is an external view of the user terminal 12, showing a GUI screen for accepting sound information. The GUI screen displays a recording acceptance icon 55. When the CPU 204 accepts a touch operation on the recording acceptance icon 55, it accepts recording via the microphone 208. The CPU 204 stores the recorded sound information, for example, in the flash memory 203, and creates sound creation parameters for designating the stored sound information.

[0040] Alternatively, the CPU 204 may display a parameter reception screen as shown in Fig. 9. Fig. 9 is an external view of the user terminal 12, showing another example of a GUI screen for receiving sound information. The GUI screen displays a parameter reception icon 57. The CPU 204 receives sound creation parameters via the parameter reception icon 57. In this case, the sound creation parameters are, for example, parameters for synthesizing the sound of a certain sound source, and parameters such as harmonic structure, pitch, envelope, filter, etc. are received from the user.

[0041] Next, the CPU 204 uses the second trained model to acquire an evaluation value of the taste information corresponding to the received sound creation parameters (S32). The CPU 204 uses the first trained model to acquire component information corresponding to the evaluation value of the acquired taste information (S33). The CPU 204 acquires product information corresponding to the acquired component information (S34). The CPU 204 inputs the component information into a database (such as the flash memory 203 or a server not shown) that specifies the component information corresponding to the product information, and acquires the corresponding product information. The CPU 204 may construct a trained model that receives component information as an input and outputs product information, and acquire the product information using the trained model.

[0042] FIG. 10 is an external view of the user terminal 12, showing an example of a product information presentation screen. The CPU 204 displays a product image 51 and a product name 52 as visual information related to the acquired product information. In the example of FIG. 10, the volume, price, and the like are also displayed as visual information. As an example, the product image 51 displays an image of a certain sake, and the product name 52 displays the name of a certain sake, Junmai Ginjo B. Such a display screen may be a screen on a mail order website. In this case, the CPU 204 may further display an order reception icon. When the CPU 204 receives a touch operation on the order reception icon, it transmits the order information to the order reception server.

[0043] In this way, the user can record a sound or input sound creation parameters to search for a product, providing a new customer experience that was not possible before.

[0044] (Variation 3) The processor according to the third modification acquires product information in which a predetermined ingredient name is equal to or less than a predetermined ingredient amount when inputting sound information and outputting product information in the processes of S33 and S34 shown in the second modification. A predetermined ingredient name being equal to or less than a predetermined ingredient amount means, for example, that the alcohol content is 0%. This allows a user who cannot consume alcohol to easily search for products that do not contain alcohol using sound. Alternatively, a user who is allergic to a specific ingredient can easily search for products that do not contain the specific allergenic ingredient using sound. Also, a user who cannot consume an ingredient for religious reasons can easily search for products that do not contain the specific ingredient using sound.

[0045] The description of the present embodiment should be considered as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

[0046] In the above embodiment, the converter is composed of the first trained model and the second trained model, and converts the product information and the sound creation parameters. However, for example, the converter may be a table that specifies the correspondence between the component information and the sensory information, and the correspondence between the sensory information and the sound information. Alternatively, the converter may be a table that specifies the correspondence between the product information including the sensory information and the sound creation parameters.

[0047] In this embodiment, taste information is shown as sensory information. However, sensory information includes smell or touch. For olfactory information, if a first trained model and a second trained model are constructed using an evaluation value, olfactory information by sound can be expressed. For tactile information, if a first trained model and a second trained model are constructed using an evaluation value, tactile information by sound can be expressed. In other words, according to this embodiment, human senses other than vision, which are difficult to convey using an information terminal, can be conveyed using sound information based on sound creation parameters.

[0048] In this embodiment, an example of sake is shown, but the alcoholic beverage can also be applied to alcoholic beverages other than sake, for example, brewed alcohol such as beer, wine, champagne, Shaoxing wine, and liqueur, and distilled alcohol such as whiskey, tequila, gin, rum, vodka, brandy, shochu, and awamori. The method of this embodiment can also be applied to non-alcoholic drinkable liquids such as soft drinks, lactic acid bacteria drinks, and milk, in addition to alcoholic beverages. The method of this embodiment can also be used for food other than beverages. In this case, the method of this embodiment can be applied by identifying the ingredient information of the food. [Explanation of symbols]

[0049] 12: User terminal 51: Product image 53: Sound icon 55: Recording reception icon 57: Parameter reception icon 71: Sake mode reception icon 72: Sake Synth mode reception icon 201:Display unit 202: User Interface 203: Flash memory 204: CPU 205:RAM 206: Communication I / F 207: Speaker 208: Mike

Claims

1. A converter is prepared that specifies the correspondence between product information, including sensory information related to taste, smell, or touch, among information related to the product, and parameters related to sound creation; inputting the product information into the converter and outputting sound information based on parameters related to sound creation, or inputting parameters related to sound creation based on the acquired sound information into the converter and outputting the product information; Information processing methods.

2. The product information includes visual information of the product and ingredient information of the product; The visual information and the component information are associated with each other, the converter defines a correspondence relationship between the component information and the sensory information, and a correspondence relationship between the sensory information and the parameters related to sound creation; receiving the visual information as the product information and outputting sound information based on the parameters relating to the sound creation, or receiving parameters relating to the sound creation based on the sound information and outputting the visual information; The information processing method according to claim 1 .

3. The sensory information includes an evaluation value by a person who has used the product, The ingredient information includes the names and amounts of ingredients analyzed for the product, The converter includes a first trained model that trains a relationship between the component name and the component amount and the evaluation value. The information processing method according to claim 2 .

4. When the sound information is input and the product information is output, product information in which a predetermined component name is equal to or less than a predetermined component amount is output. The information processing method according to claim 3.

5. The sensory information includes an evaluation value by a person who has used the product, The converter includes a second trained model that trains a relationship between the evaluation value and a parameter related to the sound creation. The information processing method according to claim 2 .

6. An evaluator evaluates the degree of agreement between the evaluation value and the sound information, and corrects the second trained model based on the evaluation. The information processing method according to claim 5.

7. the parameters relating to sound creation include information for specifying a sampled sound to be reproduced from among a plurality of sampled sounds; 7. The information processing method according to claim 1.

8. A converter is prepared that defines a correspondence between product information, including visual information about a product and sensory information about the product, such as taste, smell, or touch, and parameters related to sound creation, and the parameters related to sound creation are obtained; presenting the visual information in association with sound information based on the parameters relating to the sound creation; Product presentation method.

9. A converter is prepared that specifies the correspondence between product information, including sensory information related to taste, smell, or touch, among information related to the product, and parameters related to sound creation; inputting the product information into the converter and outputting sound information based on parameters related to sound creation, or inputting parameters related to sound creation based on the acquired sound information into the converter and outputting the product information; An information processing device having a processor.

10. The product information includes visual information of the product and ingredient information of the product; The visual information and the component information are associated with each other, the converter defines a correspondence relationship between the component information and the sensory information, and a correspondence relationship between the sensory information and the parameters related to sound creation; the processor receives the visual information as the product information and outputs sound information based on the sound creation parameters, or receives sound creation parameters based on the sound information and outputs the visual information. The information processing device according to claim 9.

11. The sensory information includes an evaluation value by a person who has used the product, The ingredient information includes the names and amounts of ingredients analyzed for the product, The converter includes a first trained model that trains a relationship between the component name and the component amount and the evaluation value. The information processing device according to claim 10.

12. the processor outputs product information in which a predetermined component name is equal to or less than a predetermined component amount when inputting a sound creation parameter based on the sound information and outputting the product information. The information processing device according to claim 11.

13. The sensory information includes an evaluation value by a person who has used the product, The converter includes a second trained model that trains a relationship between the evaluation value and the sound creation parameter. The information processing device according to claim 10.

14. The processor has an evaluator evaluate a degree of agreement between the evaluation value and the sound information, and corrects the second trained model based on the evaluation. The information processing device according to claim 13.

15. the parameters relating to sound creation include information for specifying a sampled sound to be reproduced from among a plurality of sampled sounds; 15. The information processing device according to claim 9.

16. A converter is prepared that defines a correspondence between product information, including visual information about a product and sensory information about the product, such as taste, smell, or touch, and parameters related to sound creation, and the parameters related to sound creation are obtained; presenting the visual information in association with sound information based on the parameters relating to the sound creation; A product presentation device having a processor.

17. A converter is prepared that specifies the correspondence between product information, including sensory information related to taste, smell, or touch, among information related to the product, and parameters related to sound creation; inputting the product information into the converter and outputting sound information based on parameters related to sound creation, or inputting parameters related to sound creation based on the acquired sound information into the converter and outputting the product information; A program that causes an information processing device to execute processing.

18. A converter is prepared that defines a correspondence between product information, including visual information about a product and sensory information about the product, such as taste, smell, or touch, and parameters related to sound creation, and the parameters related to sound creation are obtained; presenting the visual information in association with sound information based on the parameters relating to the sound creation; A program that causes an information processing device to execute processing.

Citation Information

Patent Citations

  • Quality prediction method for food and drink using deep learning, and food and drink

    JP2018018354A