Information processing method, product presentation method, information processing device, product presentation device, and program

The method addresses the challenge of conveying non-visual human sensations by correlating product sensory information with sound production parameters, enabling the intuitive experience of taste and sensory data through sound, thereby enhancing customer interaction with products.

WO2024241761A1PCT designated stage expired Publication Date: 2024-11-28YAMAHA CORP
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015079
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-04-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing information processing methods struggle to convey human sensations such as taste, smell, and touch through information terminals, as these senses are difficult to transmit beyond visual information.

Method used

An information processing method that utilizes a converter to establish a correspondence between product information, including sensory data like taste, smell, or touch, and sound production parameters, allowing for the output of sound information or input of sound parameters to convey product information effectively.

Benefits of technology

Enables the intuitive conveyance of human sensations beyond vision by associating visual product information with sound, allowing users to more easily imagine the taste and sensory experience of a product through synthesized sound, enhancing customer experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015079_28112024_PF_FP_ABST
    Figure JP2024015079_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This information processing method comprises: preparing a converter that defines the correspondence between product information, including sensory information relating to taste, smell, or touch that is included in information about a product, and sound production-related parameters; and outputting sound information based on sound production-related parameters by inputting product information into the converter, or outputting product information by inputting sound production-related parameters based on acquired sound information into the converter.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing method, product presentation method, information processing device, product presentation device, and program

[0001] An embodiment of the present invention relates to an information processing method, a product presentation method, an information processing device, a product presentation device, and a program.

[0002] Patent Document 1 discloses a method for predicting the quality of food and drink using deep learning.

[0003] JP 2018-18354 A

[0004] The above-mentioned prior art does not convey human senses other than sight, which is difficult to convey with an information terminal.

[0005] One aspect of the present disclosure aims to provide an information processing method that can convey human senses other than sight, which are difficult to convey using an information terminal.

[0006] An information processing method according to one embodiment of the present invention provides a converter that defines the correspondence between product information, including sensory information relating to taste, smell, or touch, and parameters related to sound creation, and inputs the product information into the converter to output sound information based on the parameters related to sound creation, or inputs the parameters related to sound creation based on acquired sound information into the converter to output the product information.

[0007] Furthermore, a product information presentation method according to one embodiment of the present invention provides a converter that defines the correspondence between product information, including visual information about a product and sensory information about the product related to taste, smell, or touch, and parameters related to sound creation, determines the parameters related to sound creation, and presents the visual information in correspondence with sound information generated based on the parameters related to sound creation.

[0008] According to one embodiment of the present invention, human senses other than sight, which are difficult to convey using an information terminal, can be conveyed using sound information based on sound creation parameters.

[0009] 1 is a block diagram showing the configuration of a user terminal 12. FIG. 2 is an external view of the user terminal 12, showing an example of a display screen (GUI) related to an application program. FIG. 3 is a flowchart showing the operation of an information processing device at the stage of constructing a trained model. FIG. 4 is a diagram showing the relationship between ingredient information, taste information, and sound creation parameters. FIG. 5 is a flowchart showing the operation of an information processing device at the stage of using a trained model. FIG. 6 is an external view of the user terminal 12, showing a display screen (GUI) related to an application program related to variant 2. FIG. 7 is a flowchart showing the operation of the Synth mode related to variant 2. FIG. 8 is an external view of the user terminal 12, showing a GUI screen for accepting sound information. FIG. 9 is an external view of the user terminal 12, showing another example of a GUI screen for accepting sound information. FIG. 10 is an external view of the user terminal 12, showing an example of a product information presentation screen.

[0010] Fig. 1 is a block diagram showing the configuration of the user terminal 12. Fig. 1 is an external view of the user terminal 12, showing an example of a display screen related to an application program.

[0011] The user terminal 12 is an information processing device such as a personal computer or a smartphone, and includes a display 201, a user I / F 202, a flash memory 203, a CPU 204, a RAM 205, a communication I / F 206, a speaker 207, and a microphone 208.

[0012] The display 201 is made up of, for example, an LCD or an OLED, and displays various information. The user I / F 202 is a touch panel that is stacked on the LCD or OLED of the display 201. Alternatively, the user I / F 202 may be a keyboard, a mouse, or the like. When the user I / F 202 is a touch panel, the user I / F 202, together with the display 201, constitutes a GUI (Graphical User Interface).

[0013] The CPU 204 is an example of a processor and is a control unit that controls the operation of the user terminal 12. The CPU 204 performs various operations by reading predetermined programs, such as application programs, stored in the flash memory 203, which is a storage medium, into the RAM 205 and executing them. The programs may be stored in a server (not shown). The CPU 204 may download the programs from the server via a network and execute them.

[0014] FIG. 2 is an external view of the user terminal 12, showing an example of a display screen (GUI) related to an application program. The CPU 204 displays a GUI screen as shown in FIG. 2 on the display 201. The GUI screen may be, for example, a screen on a mail-order website. The GUI screen displays a product image 51 and a product name 52 as visual information related to the product. In the example of FIG. 2, the visual information also displays ratings, volume, price, etc. Visual information refers to information related to the product that can be perceived visually, such as the product name, packaging, logo, product description, and advertisements. As an example, the product image 51 displays an image of a certain sake, and the product name 52 displays the name of a certain sake, such as Junmai Ginjo A. The GUI screen also displays a sound icon 53. When the CPU 204 receives a touch operation on the sound icon 53, it calculates sound creation parameters associated with the product and reproduces sound information based on the sound creation parameters. Sound information based on the sound creation parameters is different from instantaneous sounds and voice guidance sounds. For example, the CPU 204 determines parameters (such as attack, decay, and sustain) for synthesizing the sound of a certain sound source, synthesizes an analog synthesizer sound based on the parameters, and reproduces the analog synthesizer sound. That is, the CPU 204 presents visual information and sound information related to the product on the display 201 in association with each other.

[0015] The CPU 204 acquires sound information corresponding to product information from the flash memory 203 or from a server via the communication I / F 206, or acquires parameters corresponding to the product information and generates sound information based on the parameters. The flash memory 203 or the server stores sound information corresponding to each of a plurality of products. The sound information is obtained by a converter. The converter is obtained, for example, based on a trained model.

[0016] 3 is a flowchart showing the operation of the information processing device at the stage of constructing a trained model. The construction of the trained model may be performed by the CPU 204 of the user terminal 12 or by a processor of the server. Hereinafter, the operation of constructing the trained model will be referred to as being performed by a processor, even when the operation is performed by the CPU 204 of the user terminal 12.

[0017] The processor first receives product information (S11), which includes at least a product image or a product name, and then acquires ingredient information corresponding to the received product information (S12).

[0018] FIG. 4 is a diagram showing the relationship between component information, taste information, and sound creation parameters. Component information includes at least component names and component amounts. In the example of FIG. 4, the component names of a certain sake, Junmai Ginjo A, include acetaldehyde, methanol, ethyl acetate, n-propanol, i-amyl alcohol, and i-butyl alcohol, along with the respective component amounts. Component information is stored in a database, such as flash memory 203 or a server (not shown), as a result of prior product analysis. In other words, the visual information of a product and the component information of the product are associated in advance.

[0019] The processor receives taste information (S13). The taste information is an example of sensory information. As shown in FIG. 4, the taste information includes taste names such as sweetness, sourness, saltiness, bitterness, and umami, and evaluation values ​​indicating the intensity of each taste. Alternatively, the taste information may include more detailed evaluation values ​​such as fullness, richness, lightness, or smoothness. Furthermore, the taste information may include hierarchical information such as a fullness that slowly spreads in the mouth, a fullness that quickly spreads, and so on.

[0020] Such evaluation values ​​of taste information are determined based on the subjective evaluations of many users. The product in this embodiment is sake, and many users (tasters) input their impressions when tasting the sake. The many tasters taste the sake and input their taste information. As described above, each piece of product information is associated with its own component information. Therefore, by having many tasters input taste information corresponding to each piece of product information, a data set showing the relationship between the component information and the taste information is prepared. The processor trains a predetermined network model to determine the relationship between the component information and the taste information using a predetermined algorithm and the above data set (S14).

[0021] The algorithm for training the network model is not limited. The processor can train the network model using any algorithm, such as a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN). The processor may also train the network model using a hidden Markov model (HMM), a support vector machine (SVM), or the like.

[0022] This constructs a first trained model F(m)=s that takes the component information m of the product information as input and outputs an evaluation value s of the taste information.

[0023] Next, the processor accepts sound creation parameters (S15). The sound creation parameters are parameters for synthesizing the sound of a certain sound source, and include, for example, harmonic structure, pitch, envelope (time parameters such as attack, decay, sustain, and release), filter parameters, etc., as shown in Figure 4. Alternatively, the sound creation parameters may include information specifying the sampled sound data (PCM sound source) to be reproduced.

[0024] Such sound creation parameters are input, for example, by a person who specializes in sound creation (hereinafter referred to as a "sound creation craftsman"). One or more sound creation craftsmen create sound creation parameters for each of the evaluation values ​​of taste information from multiple users. Alternatively, one or more sound creation craftsmen may each sample sake and create sound creation parameters for each piece of product information. This allows a data set to be prepared that shows the relationship between taste information and sound creation parameters.

[0025] The processor trains the predetermined network model to determine the relationship between the taste information and the sound creation parameters created by the sound creator using a predetermined algorithm and the data set (S16). This constructs a second trained model G(s)=r that receives the evaluation value s of the taste information as input and outputs the sound creation parameter r.

[0026] In this way, a converter consisting of the first trained model and the second trained model, which converts product information and sound creation parameters, is prepared. The prepared converter is stored in flash memory 203 or the server.

[0027] Fig. 5 is a flowchart showing the operation of the information processing device at the stage of using the trained model. When the CPU 204 receives a touch operation of the sound icon 53 on the GUI screen shown in Fig. 2, it starts the operation of the flowchart shown in Fig. 5. First, the CPU 204 receives product information of the touched product (S21). The CPU 204 acquires ingredient information corresponding to the received product information (S22). The product information is, for example, a product name.

[0028] The CPU 204 inputs the product information into a database (such as the flash memory 203 or a server not shown) that defines the component information corresponding to the product information, and acquires the corresponding component information.

[0029] The CPU 204 may construct a trained model that receives product information as input and outputs ingredient information, and may obtain ingredient information using the trained model.

[0030] The CPU 204 uses the first trained model to acquire an evaluation value of taste information corresponding to the acquired component information (S23). The CPU 204 also uses the acquired second trained model to acquire sound creation parameters corresponding to the taste information (S24). The CPU 204 outputs sound information based on the acquired sound creation parameters (S25). For example, if the sound creation parameters are parameters for synthesizing the sound of a sound source, the CPU 204 synthesizes a sound signal from the sound source based on the parameters and outputs the synthesized sound signal. If the sound creation parameters are information specifying sound data to be played (PCM sound source), the CPU 204 plays the specified sound data and outputs it as a sound signal. The sound signal output by the CPU 204 is emitted, for example, from the speaker 207. That is, in this embodiment, the CPU 204 prepares a converter that specifies the correspondence between product information, including sensory information related to taste among information about the product, and sound creation parameters, inputs the product information into the converter, and outputs sound information based on the sound creation parameters. The CPU 204 also prepares a converter that defines the correspondence between product information, including visual information about the product and sensory information about the taste of the product, and parameters related to sound creation, determines the parameters related to sound creation, and presents the visual information in correspondence with sound information based on the sound creation parameters on a GUI screen such as that shown in Figure 2.

[0031] The user then hears the sound emitted from the speaker 207. The sound is expertly crafted to correlate with sensory information related to taste. Sound has multidimensional elements, such as time variations and frequency components, and a few seconds of sound produces information that changes along the time axis, allowing for the conveyance of more information. Expressing taste through sound in this way can more intuitively evoke taste than visually expressing the intensity of sweetness, sourness, etc., using numerical values. For example, a product with a strong sweet taste is emitted as a full, rounded, and rich sound with suppressed high frequencies, a sound that is highly correlated with the intensity of sweetness. A product with a strong smooth taste is expressed as a smooth sound with few odd-order harmonics. For example, a feeling of fullness that slowly spreads in the mouth is expressed as a strong low-frequency sound, while a feeling of fullness that quickly spreads is expressed as a strong high-frequency sound. Therefore, the user can easily imagine the taste of the product by listening to the sound, which is a new customer experience that was not possible before.

[0032] (Variation 1) A processor according to Variation 1 receives the degree of match between the evaluation value and sound information from an evaluator, and corrects the first trained model based on the received evaluation. The degree of match is determined based on the evaluator's subjective evaluation. The evaluator may be a user of the product (a taster in the above embodiment) or a sound craftsman. The evaluator listens to a sound based on sound creation parameters calculated using the second trained model G(s) = r, and inputs the degree of match ω with the taste information. The processor uses the received degree of match ω to minimize the error L between the second trained model G(s) and the output value r, as shown in the following formula (1).

[0033]

[0034] In Equation (1), ωn is the degree of agreement between the evaluation value s of the nth product and the sound creation parameter r. The second trained model G(s) is further optimized by having multiple evaluators input the degree of agreement ω.

[0035] This allows users to have a new customer experience that was not possible before: they can more easily imagine the taste of the product by listening to the sound.

[0036] (Variant 2) The processor according to variant 2 uses a converter that defines the correspondence between product information, including sensory information relating to taste, smell, or touch, among information about the product, and parameters related to sound creation, and inputs parameters related to sound creation based on the acquired sound information into the converter and outputs the product information.

[0037] 6 is an external view of the user terminal 12, showing a display screen (GUI) for an application program according to Modification 2. The CPU 204 displays a GUI screen as shown in Fig. 6 on the display 201. The GUI screen displays a Kikizake mode reception icon 71 and a Sake Synth mode reception icon 72.

[0038] When the CPU 204 receives a touch operation on the listening sake mode reception icon 71, it executes the product presentation method of the flowchart shown in Fig. 5. On the other hand, when the CPU 204 receives a touch operation on the sake synth mode reception icon 72, it executes the processing of the flowchart shown in Fig. 7. Fig. 7 is a flowchart showing the operation of the sake synth mode according to Modification 2. The sake synth mode is a mode in which, by inputting sound information, information on the corresponding product (sake in this embodiment) is presented.

[0039] First, the CPU 204 accepts sound information (S31). Fig. 8 is an external view of the user terminal 12, showing a GUI screen for accepting sound information. The GUI screen displays a recording acceptance icon 55. When the CPU 204 accepts a touch operation on the recording acceptance icon 55, it accepts recording via the microphone 208. The CPU 204 stores the recorded sound information, for example, in the flash memory 203, and creates sound creation parameters for specifying the stored sound information.

[0040] Alternatively, the CPU 204 may display a parameter reception screen as shown in Fig. 9. Fig. 9 is an external view of the user terminal 12, showing another example of a GUI screen for receiving sound information. The GUI screen displays a parameter reception icon 57. The CPU 204 receives sound creation parameters via the parameter reception icon 57. In this case, the sound creation parameters are, for example, parameters for synthesizing the sound of a certain sound source, and parameters such as harmonic structure, pitch, envelope, filter, etc. are received from the user.

[0041] Next, the CPU 204 uses the second trained model to acquire an evaluation value of the taste information corresponding to the received sound creation parameters (S32). The CPU 204 uses the first trained model to acquire ingredient information corresponding to the evaluation value of the acquired taste information (S33). The CPU 204 acquires product information corresponding to the acquired ingredient information (S34). The CPU 204 inputs the ingredient information into a database (such as the flash memory 203 or a server not shown) that defines ingredient information corresponding to the product information, and acquires the corresponding product information. Note that the CPU 204 may construct a trained model that inputs ingredient information and outputs product information, and acquire the product information using the trained model.

[0042] FIG. 10 is an external view of the user terminal 12, showing an example of a product information presentation screen. The CPU 204 displays a product image 51 and a product name 52 as visual information related to the acquired product information. In the example of FIG. 10, the volume, price, etc. are also displayed as visual information. As an example, the product image 51 displays an image of a certain sake, and the product name 52 displays the name of a certain sake, Junmai Ginjo B. Such a display screen may be a screen on a mail-order website. In this case, the CPU 204 may further display an order reception icon. When the CPU 204 receives a touch operation on the order reception icon, it transmits order information to the order reception server.

[0043] In this way, users can record sounds or input sound creation parameters to search for products, providing a new customer experience that was not possible before.

[0044] (Variation 3) In the processing of S33 and S34 shown in Variation 2, the processor according to Variation 3 acquires product information in which a predetermined ingredient name is equal to or less than a predetermined ingredient amount when inputting sound information and outputting product information. A predetermined ingredient name equal to or less than a predetermined ingredient amount means, for example, that the alcohol content is 0%. This allows a user who cannot consume alcohol to easily use sound to search for products that do not contain alcohol. Alternatively, a user who is allergic to a specific ingredient can easily use sound to search for products that do not contain the specific allergen. Furthermore, a user who cannot consume an ingredient for religious reasons can easily use sound to search for products that do not contain the specific ingredient.

[0045] The description of the present embodiment should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

[0046] In the above embodiment, the converter includes a first trained model and a second trained model, and converts product information and sound creation parameters. However, for example, the converter may be a table that defines the correspondence between component information and sensory information, and the correspondence between sensory information and sound information. Alternatively, the converter may be a table that defines the correspondence between product information including sensory information and sound creation parameters.

[0047] In this embodiment, taste information is shown as sensory information. However, sensory information also includes olfactory information and tactile information. For olfactory information, if a first trained model and a second trained model are constructed using evaluation values, olfactory information can be expressed by sound. For tactile information, if a first trained model and a second trained model are constructed using evaluation values, tactile information can be expressed by sound. In other words, according to this embodiment, human senses other than sight, which are difficult to convey using an information terminal, can be conveyed using sound information based on sound creation parameters.

[0048] Although sake is used as an example in this embodiment, the method can also be applied to alcoholic beverages other than sake, such as brewed alcoholic beverages such as beer, wine, champagne, Shaoxing wine, and liqueurs, and distilled alcoholic beverages such as whiskey, tequila, gin, rum, vodka, brandy, shochu, and awamori. The method of this embodiment can also be applied to non-alcoholic beverages, such as coffee, tap water, natural spring water, black tea, green tea, oolong tea, barley tea, dokudami tea, and sweet tea, soft drinks, lactic acid bacteria drinks, and milk. The method of this embodiment can also be used for food in addition to beverages. In this case, the method of this embodiment can be applied by identifying the ingredient information of the food.

[0049] 12: User terminal, 51: Product image, 53: Sound icon, 55: Recording reception icon, 57: Parameter reception icon, 71: Sake mode reception icon, 72: Sake Synth mode reception icon, 201: Display, 202: User I / F, 203: Flash memory, 204: CPU, 205: RAM, 206: Communication I / F, 207: Speaker, 208: Microphone

Claims

1. An information processing method comprising: preparing a converter that specifies the correspondence between product information, including sensory information relating to taste, smell or touch, and parameters relating to sound creation; inputting the product information into said converter and outputting sound information based on the parameters relating to sound creation; or inputting the parameters relating to sound creation based on acquired sound information into said converter and outputting the product information.

2. The information processing method of claim 1, wherein the product information includes visual information of the product and ingredient information of the product, the visual information and the ingredient information are associated with each other, the converter defines a correspondence between the ingredient information and the sensory information, and a correspondence between the sensory information and parameters related to the sound creation, and accepts the visual information as the product information and outputs sound information based on the parameters related to the sound creation, or accepts parameters related to sound creation based on the sound information and outputs the visual information.

3. The information processing method of claim 2, wherein the sensory information includes an evaluation value by a person who has used the product, the ingredient information includes names and amounts of ingredients analyzed from the product, and the converter includes a first trained model that has trained the relationship between the ingredient names and amounts of ingredients and the evaluation value.

4. The information processing method according to claim 3, further comprising the step of: inputting said sound information and outputting said product information, and outputting product information in which a predetermined component name is equal to or less than a predetermined component amount.

5. An information processing method as described in claim 2, wherein the sensory information includes an evaluation value by a person who has used the product, and the converter includes a second trained model that trains the relationship between the evaluation value and parameters related to sound creation.

6. The information processing method according to claim 5, further comprising the steps of: an evaluator evaluates the degree of match between the evaluation value and the sound information; and correcting the second trained model based on the evaluation.

7. An information processing method according to any one of claims 1 to 6, wherein the parameters relating to sound creation include information for specifying which of a plurality of sampled sounds is to be reproduced.

8. A product presentation method comprising: preparing a converter that defines the correspondence between product information, including visual information about a product and sensory information about the product related to taste, smell or touch, and parameters related to sound creation, determining the parameters related to sound creation, and presenting the visual information in correspondence with sound information based on the parameters related to sound creation.

9. An information processing device having a processor, which prepares a converter that specifies the correspondence between product information, including sensory information related to taste, smell or touch, and parameters related to sound creation, and inputs the product information into the converter and outputs sound information based on the parameters related to sound creation, or inputs the parameters related to sound creation based on acquired sound information into the converter and outputs the product information.

10. The information processing device of claim 9, wherein the product information includes visual information of the product and ingredient information of the product, the visual information and the ingredient information are associated with each other, the converter defines a correspondence between the ingredient information and the sensory information, and a correspondence between the sensory information and parameters related to the sound creation, and the processor accepts the visual information as the product information and outputs sound information based on the sound creation parameters, or accepts sound creation parameters based on the sound information and outputs the visual information.

11. The information processing device of claim 10, wherein the sensory information includes an evaluation value by a person who has used the product, the ingredient information includes names and amounts of ingredients analyzed from the product, and the converter includes a first trained model that trains the relationship between the ingredient names and amounts of ingredients and the evaluation value.

12. The information processing device according to claim 11, wherein said processor outputs product information in which a predetermined component name is equal to or less than a predetermined component amount when inputting sound creation parameters based on said sound information and outputting said product information.

13. The information processing device according to claim 10, wherein the sensory information includes an evaluation value by a person who has used the product, and the converter includes a second trained model that trains the relationship between the evaluation value and the sound creation parameters.

14. The information processing device according to claim 13, wherein the processor has an evaluator evaluate the degree of match between the evaluation value and the sound information, and corrects the second trained model based on the evaluation.

15. An information processing apparatus according to any one of claims 9 to 14, wherein the parameters relating to sound creation include information for specifying which of a plurality of sampled sounds is to be reproduced.

16. A product presentation device equipped with a processor, which prepares a converter that defines the correspondence between product information, including visual information about a product and sensory information about the product related to taste, smell or touch, and parameters related to sound creation, determines the parameters related to sound creation, and presents the visual information in correspondence with sound information based on the parameters related to sound creation.

17. A program that causes an information processing device to execute a process of preparing a converter that specifies the correspondence between product information, including sensory information related to taste, smell or touch, and parameters related to sound creation, and inputting the product information into the converter and outputting sound information based on the parameters related to sound creation, or inputting the parameters related to sound creation based on acquired sound information into the converter and outputting the product information.

18. A program that causes an information processing device to execute a process of preparing a converter that specifies the correspondence between product information, including visual information about a product and sensory information about the product related to taste, smell or touch, and parameters related to sound creation, determining the parameters related to sound creation, and presenting the visual information in correspondence with sound information based on the parameters related to sound creation.

Citation Information

Patent Citations

  • Information conversion device, method, and program

    JP2016076118A

  • Sound symbolic word and taste impression information providing device, sound symbolic word and taste impression information providing method, and sound symbolic word and taste impression information providing program

    JP2017208019A

  • Sensory information presentation system and sensory information presentation method

    JP2018073019A

  • Brewage retrieval device, and program

    JP2019128895A

  • Commodity development support system, commodity development support method, and program

    JP2020140569A