Information Processing System

Abstract

The present invention provides an information processing system, an information processing device, and an information processing method. The present invention comprises a display means, a wearable device provided with an image capture means, and a database connected to the wearable device via a network, wherein: the database retains information relating to cooking recipes, cooking methods, and / or ingredients; the wearable device detects a first ingredient by means of the image capture means; and the wearable device collects information relating to the first ingredient from the database, and displays the information relating to the first ingredient on the display means if the first ingredient is present in a specific region of an area, an image of which is captured by the image capture means, or does not display the information relating to the first ingredient on the display means if the first ingredient is not present in the specific region.

Description

[Technical field]

[0001] One aspect of the present invention relates to an information processing system, an information processing device, and an information processing method. Also, one aspect of the present invention relates to a cooking assistance system, a cooking assistance device, and a cooking assistance method. [Background technology]

[0002] In recent years, cooking recipe introduction services via information terminals such as smartphones and tablets have been gaining popularity (Patent Document 1).

[0003] Users can keep the information terminal with the recipe displayed on it close at hand and cook while referring to the recipe.

[0004] Furthermore, in recent years, a technique for detecting objects from an image using a neural network has been proposed.

[0005] The object detection is a technology that extracts a bounding box from an image where an object is expected to be captured, and recognizes the object within the bounding box (Patent Document 2). R-CNN (Regions with Convolutional Neural Networks), YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and the like have been proposed as methods for the object detection.

[0006] In addition, in recent years, a technology called semantic segmentation has been proposed that uses a neural network to divide an image into regions and assign a label to each divided region (Patent Document 3).

[0007] Methods proposed for the above semantic segmentation include FCN (Fully Convolutional Network), SegNet, U-Net, and PSPNet (Pyramid Scene Parsing Network). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2018-124672 A [Patent Document 2] JP 2017-55177 A [Patent Document 3] Special Publication No. 2018-535491 Summary of the Invention [Problem to be solved by the invention]

[0009] Information terminals such as smartphones and tablets are generally operated using touch panels. On the other hand, when a user cooks while looking at such an information terminal, it is necessary to operate the information terminal according to the cooking process. However, if the user has water, ingredients, seasonings, etc. on their hands while cooking, they may not be able to operate the information terminal correctly, which is inconvenient. In addition, touching the information terminal with such hands may cause the information terminal to become dirty, and may even cause the information terminal to break down due to the dirt, which is undesirable. In addition, continuing cooking with hands that have touched an information terminal is undesirable from the standpoint of food hygiene.

[0010] In addition, kitchens are equipped with sources of fire such as water and stoves, so bringing an information terminal into a kitchen may cause the information terminal to malfunction due to water or fire. Also, if an electromagnetic cooker is installed in a kitchen, the information terminal may malfunction or malfunction due to electromagnetic fields.

[0011] An object of one embodiment of the present invention is to provide an information processing system that allows a user to obtain information without using hands. Another object of one embodiment of the present invention is to provide an information processing device that allows a user to obtain information without using hands. Another object of one embodiment of the present invention is to provide an information processing method that allows a user to obtain information without using hands. Another object of one embodiment of the present invention is to provide a cooking assistance system that allows a user to obtain information without using hands. Another object of one embodiment of the present invention is to provide a cooking assistance device that allows a user to obtain information without using hands. Another object of one embodiment of the present invention is to provide a cooking assistance method that allows a user to obtain information without using hands. [Means for solving the problem]

[0012] One embodiment of the present invention is an information processing system having a wearable device with a display means and an imaging means, and a database connected to the wearable device via a network, the database having at least one of information regarding a cooking recipe, a cooking method, and ingredients, the wearable device detects a first ingredient using the imaging means, collects information regarding the first ingredient from the database, and when the first ingredient is present in a specific area within the range imaged by the imaging means, displays the information regarding the first ingredient on the display means, and when the first ingredient is not present in the specific area, hides the information regarding the first ingredient from the display means.

[0013] In the above, it is preferable that a cooking method using the first ingredient is displayed on the display means based on a cooking recipe.

[0014] In the above, the information about the first material preferably includes a cutting position of the first material.

[0015] In the above, the information about the first material preferably includes positions of bones contained in the first material.

[0016] In the above, the wearable device is preferably a glasses-type wearable device.

[0017] In the above, the database is preferably stored on a server.

[0018] One embodiment of the present invention is an information processing system having a wearable device with a display means and an imaging means, and a cooking utensil having a temperature sensor, wherein the wearable device and the temperature sensor are connected via a first network, the wearable device detects the cooking utensil using the imaging means, the wearable device collects information about the temperature inside the cooking utensil from the temperature sensor, and when the cooking utensil is present in a specific area within the range imaged by the imaging means, displays the information about the temperature on the display means, and when the cooking utensil is not present in the specific area, hides the information about the temperature from the display means.

[0019] In the above, it is preferable that the information processing system further has a database, the database being connected to the wearable device and the temperature sensor via a second network including the first network, the database receiving information regarding temperature via the second network, and the database calculating the time required to heat the cooking utensil from the information regarding temperature and displaying it on the display means.

[0020] In the above, the wearable device is preferably a glasses-type wearable device.

[0021] In the above, the database is preferably stored on a server.

[0022] One aspect of the present invention is an information processing method using a wearable device having a display means and an imaging means, where the wearable device is worn by a user so that the user can see ingredients and cooking utensils through the display means, and the information processing method includes the steps of detecting a cutting board present in the user's line of sight using the imaging means, identifying a first ingredient placed on the cutting board, displaying a cooking method on the display means, and displaying a cutting position of the first ingredient on the display means so as to overlap with the first ingredient in the user's line of sight.

[0023] In the above, it is preferable that the information processing method further includes a step of displaying the position of a foreign object present on the surface or inside of the material on the display means so as to overlap with the material in the user's line of sight.

[0024] In the above, the foreign matter is preferably one selected from bones, scales, parasites, and hair.

[0025] In the above, the wearable device is preferably a glasses-type wearable device. Effect of the Invention

[0026] According to one aspect of the present invention, it is possible to provide an information processing system that allows information to be obtained without using hands. According to another aspect of the present invention, it is possible to provide an information processing device that allows information to be obtained without using hands. According to another aspect of the present invention, it is possible to provide an information processing method that allows information to be obtained without using hands. According to another aspect of the present invention, it is possible to provide a cooking assistance system that allows information to be obtained without using hands. According to another aspect of the present invention, it is possible to provide a cooking assistance device that allows information to be obtained without using hands. According to another aspect of the present invention, it is possible to provide a cooking assistance method that allows information to be obtained without using hands. [Brief description of the drawings]

[0027] FIG. 1 is a schematic diagram illustrating an example of a method of using the AR glasses according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating one embodiment of the present invention. FIG. 3 is a flowchart illustrating one embodiment of the present invention. 4A and 4B are schematic diagrams illustrating one embodiment of the present invention. 5A and 5B are schematic diagrams illustrating one embodiment of the present invention. 6A and 6B are schematic diagrams illustrating one embodiment of the present invention. FIG. 7 is a schematic diagram for explaining an example of a method for displaying a guideline according to an embodiment of the present invention. Fig. 8A is a schematic diagram illustrating a method for training a neural network to learn training data using machine learning, which is one aspect of the present invention. Fig. 8B is a schematic diagram illustrating a method for generating an image using a trained neural network, which is one aspect of the present invention. FIG. 9 is a schematic diagram illustrating a method for generating an image according to one embodiment of the present invention. Fig. 10A is a schematic diagram illustrating a method for training a neural network to learn training data using machine learning, which is one aspect of the present invention. Fig. 10B is a schematic diagram illustrating a method for generating an image using a trained neural network, which is one aspect of the present invention. 11A and 11B are schematic diagrams illustrating one embodiment of the present invention. 12A and 12B are schematic diagrams illustrating an example of a method for displaying a guide line according to an embodiment of the present invention. 13A and 13B are schematic diagrams illustrating one embodiment of the present invention. 14A and 14B are schematic diagrams illustrating a cooking utensil detection method according to one embodiment of the present invention. 15A and 15B are schematic diagrams illustrating one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] (Embodiment 1) In this embodiment, an information processing system according to one embodiment of the present invention will be described. The information processing system according to this embodiment can assist a user in cooking.

[0029] In the information processing system of this embodiment, the user can perform cooking while using an information terminal. As the information terminal, it is preferable that the user can wear it. In this embodiment, an example of using a wearable device as the information terminal used by the user is shown. Furthermore, the wearable device shown in this embodiment is preferably a glasses-type wearable device, and such a wearable device may be called AR glasses in some cases.

[0030] <Examples of using AR glasses> FIG. 1 shows an example of a user performing cooking while wearing AR glasses 10. Note that AR is an abbreviation for Augmented Reality (augmented reality), and the AR glasses 10 can project information such as images and characters onto the external world that the user is looking at.

[0031] The AR glasses 10 can present information necessary for the user to cook, such as cooking recipes, cooking methods, and information about ingredients. Here, ingredients include plant-based ingredients such as grains, vegetables, fruits, and seaweeds, animal-based ingredients such as seafood, meat, eggs, dairy products, and bones, seasonings, spices, oils, food additives, and other ingredients necessary for cooking. Also, ingredients may be referred to as food ingredients in some cases. The user can obtain the information necessary for cooking through the AR glasses 10 even when both hands are occupied.

[0032] The AR glasses 10 are composed of a glass part 10a, a housing 10b, and a wiring 10c. The wiring 10c connects the glass part 10a and the housing 10b. In FIG. 1, the housing 10b is stored in the pocket of the apron. Note that when the glass part 10a and the housing 10b can communicate wirelessly, the wiring 10c may be omitted. Also, when all the necessary elements of the AR glasses 10 fit within the glass part 10a, the housing 10b and the wiring 10c may be omitted.

[0033] <Configuration example of AR glasses> 2 is a block diagram showing an example of the configuration of the AR glasses 10. The AR glasses 10 include a first camera module 11, a second camera module 12, a controller 13, a display unit 14, a processor 15, a memory 16, a communication module 17, an audio controller 18, a microphone 19, a speaker 20, a battery 21, a sensor 25, and a bus 22.

[0034] The first camera module 11, the second camera module 12, the controller 13, the processor 15, the memory 16, the communication module 17, the sensor 25, and the audio controller 18 exchange data via a bus 22.

[0035] The first camera module 11 has a function of acquiring the user's visual field as an image, and the second camera module 12 has a function of capturing the movement of the user's eyeballs and detecting the user's line of sight.

[0036] Although details will be described later, in order to display various information on the display unit 14 according to the image captured by the first camera module 11, it is preferable that the imaging direction of the first camera module 11 roughly coincides with the line of sight of the user. Also, it is preferable that the imaging range of the first camera module 11 includes the field of view seen by the user through the AR glasses 10. In other words, it is preferable that a specific area of ​​the imaging range of the first camera module 11 coincides with the field of view seen by the user through the AR glasses 10.

[0037] The audio controller 18 has a function of analyzing an audio signal acquired by the microphone 19 and converting it into a digital signal, and also has a function of generating an audio signal to be outputted from the speaker 20.

[0038] The controller 13 has a function of generating an image to be displayed on the display unit 14 .

[0039] The communication module 17 has a function of communicating with the database via a network such as the Internet. This allows the AR glasses 10 to communicate with the database via the network. Data downloaded from the database is stored in the memory 16. The database is preferably stored in a server 23. In this case, the AR glasses 10 connect to the server 23 via the network using the communication module 17. Alternatively, the AR glasses 10 may connect to the server 23 via a device connected to the AR glasses 10 and the server 23 via the network. As the device, a desktop computer 26, a laptop computer 27, a tablet computer 28 (tablet terminal), a smartphone 29, or the like can be used.

[0040] The database may be stored in the device. The user can use the device to register user information, set usage conditions, and register cooking utensils used by the user in the AR glasses 10 and the database.

[0041] For example, it is preferable that the server 23 performs machine learning using training data. The AR glasses 10 can download the learning results obtained by the above learning from the server 23 and store them in the memory 16.

[0042] The battery 21 has a function of supplying power to each device constituting the AR glasses 10.

[0043] Processor 15 has a function of comprehensively controlling devices connected to bus 22. For example, processor 15 comprehensively judges information acquired from first camera module 11, second camera module 12, communication module 17, and audio controller 18, and issues instructions to controller 13. In response to instructions from processor 15, controller 13 generates image data and displays it on display unit 14.

[0044] It is preferable to use a so-called see-through panel that transmits external light as the display unit 14. Examples of the see-through panel include an organic EL (Electro Luminescence) display and a liquid crystal display.

[0045] The higher the definition of the see-through panel, the more preferable. The pixel density of the see-through panel can be 1000ppi or more and 50000ppi or less, preferably 2000ppi or more and 50000ppi or less, more preferably 3000ppi or more and 50000ppi or less, and even more preferably 5000ppi or more and 50000ppi or less. Typically, the pixel density can be 4500ppi or more and 5500ppi or less, 5500ppi or more and 6500ppi or less, or 6500ppi or more and 7500ppi or less.

[0046] Moreover, the higher the resolution of the see-through panel, the more preferable. The number of pixels of the see-through panel in the scanning line direction or signal line direction can be, for example, 1000 to 20000, preferably 2000 to 20000, more preferably 3000 to 20000. When two see-through panels are provided for the left eye and the right eye, the shape of the display area can be close to a square (the ratio of the length to the width is 0.8 to 1.2). On the other hand, when one display area is used separately for the right eye and the left eye, the shape of the display area is preferably a horizontally long rectangle (for example, the ratio of the length to the width is 1.5 to 5.0). Moreover, the see-through panel may be adapted to the television standard with an aspect ratio of 16:9, in which case the resolution can be the FHD standard, the 4K2K standard, or the 8K4K standard.

[0047] The display unit 14 may also have a device for projecting an image onto a reflector placed in front of the user's eyes. In this case, the display unit 14 has optical members such as a light guide plate and a half mirror, and a light-emitting element. Examples of the light-emitting element include an organic EL element, an LED (Light Emitting Diode) element, and an inorganic EL element.

[0048] The display unit 14 may also have a retinal projection display device. A retinal projection display device is a device that projects an image onto a user's retina by irradiating the retina with a weak laser light. In this case, the display unit 14 has a laser oscillator, an optical system (light guide plate, reflector, half mirror, etc.), etc. In addition, the laser oscillator is preferably controlled by a MEMS (Micro Electro Mechanical System).

[0049] A user wearing the AR glasses 10 can see ingredients and utensils necessary for cooking through the display unit 14. Information about the ingredients and cooking utensils that the user is looking at is displayed on the display unit 14 in the user's line of sight. In other words, when viewed from the user's perspective, information about the ingredients and cooking utensils is displayed on the display unit 14 so as to overlap with the ingredients and cooking utensils.

[0050] The AR glasses 10 may also have other sensors 25, such as an acceleration sensor, a gyro sensor, a temperature sensor, and an electrooculography sensor. The electrooculography sensor is a sensor that detects a change in potential caused by eye movement, and is provided in the glass part 10a. By having the electrooculography sensor, the AR glasses 10 can analyze the movement of the user's eyes and track the user's line of sight.

[0051] 2, the AR glasses 10 may include the processor 15, memory 16, communication module 17, and battery 21 in the housing 10b, and the other components in the glass section 10a. This makes it possible to reduce the weight of the glass section 10a, thereby reducing the burden felt by the user.

[0052] 2, only the battery 21 may be included in the housing 10b, and the other components may be included in the glass section 10a. By arranging the devices connected to the bus 22 nearby, it is possible to improve the processing speed and save power.

[0053] Furthermore, if it is possible to further reduce the size of the battery 21, all of the structural elements of the AR glasses 10 may be included in the glass part 10a. In that case, the housing 10b and the wiring 10c are not necessary.

[0054] <System details> The information processing system using the AR glasses 10 will be described in detail with reference to Figures 3 to 10. In the following description, the information processing system will be simply abbreviated to "system".

[0055] Fig. 3 is a flow chart for explaining the flow of processing of the system. Fig. 3 includes steps S0 to S10. Details of the system will be explained below in order starting from step S0. Note that in this embodiment, a case where fish is prepared using the AR glasses 10 will be explained.

[0056] First, in step S0, the user starts the system. A field of view 50 shown in FIG. 4A represents a field of view that the user can see through the AR glasses 10 immediately after the system is started. In FIG. 4A, a cutting board 60 and ingredients 62, as well as a knife 63 and a dish towel 64, exist in the field of view 50. In this embodiment, an example is shown in which a fish is used as the ingredients 62. Note that the imaging range of the first camera module 11 preferably includes the field of view 50. In other words, it is preferable that a specific area of ​​the imaging range of the first camera module 11 coincides with the field of view 50.

[0057] It is preferable that the cutting board 60 is a plain board with few stains or scratches. The cutting board 60 has markers 61a, 61b, 61c, and 61d arranged at its four corners.

[0058] The markers 61a to 61d may be printed directly on the cutting board 60, or may be attached to the cutting board by the user like stickers. The number of markers is not limited to this, and for example, the markers may be placed only on the three corners of the cutting board 60.

[0059] Next, in step S1, the system detects the markers 61a to 61d using the first camera module 11 to detect the position of the cutting board 60. The system adds only the area surrounded by the markers 61a to 61d to the analysis target, and excludes the area outside of the area from the analysis target. For example, as shown in FIG. 4B, the system excludes the area indicated by the diagonal lines in the figure from the analysis target.

[0060] The system attempts to recognize the ingredients 62 by image analysis, but if objects around the cutting board 60, such as the knife 63 and the dish towel 64, are included in the image analysis, this adversely affects the accuracy of the recognition of the ingredients 62. As shown in FIG. 4B, the system can improve the accuracy of the recognition of the ingredients 62 by excluding objects around the cutting board 60 from the analysis target. In addition, by narrowing the area of ​​the image to be analyzed, it becomes possible to perform image analysis at a higher speed.

[0061] In addition, without using markers, the area of ​​the cutting board 60 may be detected using object detection methods such as R-CNN, YOLO, and SSD, as described in the background art, or semantic segmentation methods such as FCN, SegNet, U-Net, and PSPNet.

[0062] Next, in step S2, the system identifies the type of ingredient 62. As an example, in Fig. 5A, a rectangle 65 is displayed surrounding the ingredient 62, and candidate ingredients 62 are displayed in a menu 70. The above-mentioned object detection method is preferably used to identify the ingredient 62. The menu 70 preferably displays candidate ingredients 62 in order of likelihood.

[0063] The user can select the type of ingredient 62 from the candidates displayed in the menu 70. The user can select the menu by voice input. For example, when the user utters "horse mackerel," the system analyzes the user's voice and determines that "horse mackerel" has been selected.

[0064] The user can also select a menu by line of sight. For example, the user stares at the "horse mackerel" item displayed on the menu 70. The AR glasses 10 detect the user's line of sight and determine that "horse mackerel" has been selected. The user's line of sight can be detected using the second camera module 12. The user's line of sight can be detected by analyzing the positional relationship between the inner corner of the user's eye and the iris from the image of the user's eyeball acquired by the second camera module 12, using the inner corner of the user's eye as a reference point.

[0065] Furthermore, the user's line of sight can be detected by mounting an infrared source such as an infrared LED and an infrared detector such as an infrared camera on the AR glasses 10. The user's line of sight can be detected by irradiating the user's eye with infrared light emitted from the infrared source, using the reflection position on the cornea (called the corneal reflex) as the reference point, and analyzing the positional relationship between the corneal reflex and the pupil.

[0066] Furthermore, if the AR glasses 10 have an electrooculography sensor, the AR glasses 10 may track the user's line of sight by analyzing the user's electrooculography.

[0067] At this time, a pointer 51 may be displayed on the menu 70 in accordance with the user's line of sight. By displaying the pointer 51, the user can determine whether the item that the user intended has been selected.

[0068] The pointer 51 may be displayed only when an item is selected from the menu 70. Alternatively, the sensor 25 may be used to detect the movement of the user and display or hide the pointer 51. For example, the pointer 51 may be displayed or hidden by the user wearing the AR glasses 10 shaking his / her head up and down or left and right. As a more specific example, the pointer 51 can be displayed by the user wearing the AR glasses 10 shaking his / her head up and down, and the pointer 51 can be hidden by shaking his / her head left and right.

[0069] In this way, it is preferable for the user to be able to make menu selections without using their hands, allowing the user to operate the system even when their hands are full while cooking.

[0070] Meanwhile, the first camera module 11 may be used to detect the movement of the user's hand. The user can select a desired item on the display unit 14 by placing their hand, particularly their fingers, so that the hand overlaps with the desired item on the menu 70. In this case, the selected item may be determined in combination with a known gesture recognition method, such as a finger gesture.

[0071] The AR glasses 10 read the user's movements, allowing the user to operate the system without touching the information terminal.

[0072] As described above, the user can operate the system without touching the information terminal, which can prevent the information terminal from breaking down or malfunctioning due to touching the information terminal with wet hands or hands with ingredients on them. In addition, there is no need to continue cooking with hands that have touched the information terminal, which is preferable from the standpoint of food hygiene.

[0073] It is preferable that the menu 70 and the pointer 51 are displayed when the ingredient 62 is present in the field of view 50. If the ingredient 62 is no longer present in the field of view 50 due to the movement of the user's line of sight or the movement of the ingredient 62, the menu 70 and the pointer 51 may be hidden. Alternatively, the menu 70 and the pointer 51 may be hidden after a certain period of time has passed in which the ingredient 62 is not present in the field of view 50.

[0074] Next, in step S3, the system displays recipe candidates using ingredient 62. In FIG. 5B, as an example, recipe candidates using horse mackerel are displayed in menu 70. The user can select a recipe from menu 70 in the same manner as in step S2. It is preferable that the recipe display health-related information such as calories, salt content, and carbohydrate content, as well as necessary ingredients and seasonings. In addition, when selecting a recipe, the user can register the number of people for whom the dish will be served, taste preferences, etc.

[0075] Next, in step S4, the system displays the recipe selected in step S3. In Fig. 6A, recipe 71 is displayed in the upper part of field of view 50 as an example.

[0076] Recipe 71 displays the number of each step and an image to explain each step. When a step in recipe 71 is selected in the same way as in step S2, an image of the selected step is displayed large (FIG. 6B). In addition, text 72 explaining the step and text 73 explaining useful advice are displayed below field of view 50. This allows the user to understand important points for each step. Note that the images displayed in recipe 71 are not limited to still images, and moving images may also be used.

[0077] It is preferable that recipe 71, text 72, text 73, and pointer 51 are displayed when either ingredients 62 or cutting board 60 is present in field of view 50. When ingredients 62 and cutting board 60 are no longer present in field of view 50 due to the movement of the user's line of sight or the movement of ingredients 62 or cutting board 60, recipe 71, text 72, text 73, and pointer 51 may be hidden. Alternatively, recipe 71, text 72, text 73, and pointer 51 may be hidden after a certain period of time has passed in which ingredients 62 and cutting board 60 are not present in field of view 50.

[0078] Next, in step S5, the user selects whether to enable or disable the display function of the knife guideline, which indicates the cutting position of the material. If the knife guideline is enabled, in step S6, the system displays the knife guideline, which indicates the appropriate cutting position of the material.

[0079] FIG. 7 shows what the field of view 50 looks like when knife guidelines are displayed. The user is filleting a fish (ingredient 62) with a knife. At this time, the system displays guidelines 75 so that they overlap with the ingredient 62. The user can correctly cut and cook the ingredient 62 by inserting the blade of the knife along the guidelines 75. By displaying the guidelines 75 three-dimensionally in line with the shape of the ingredient 62, the user can easily cut and process the ingredient 62.

[0080] A method for displaying the knife guideline in the system will be described with reference to Figures 8A and 8B. It is preferable that the knife guideline is displayed using machine learning using a neural network.

[0081] 8A shows a method for training a neural network using training data. The training is preferably performed by the server 23 shown in FIG.

[0082] Fig. 8B shows a method for generating a target image using a trained neural network. The process shown in Fig. 8B is performed inside the AR glasses 10. The AR glasses 10 download the trained neural network from the server 23 and generate a target image.

[0083] First, the learning method of Fig. 8A will be explained. A data set 80 is prepared as training data. The data set 80 has an image set 81 and an image set 82. The image set 81 has images of a plurality of materials. The image set 82 has a start point 83 represented by a circle and an end point 84 also represented by a circle added to each image in the image set 81.

[0084] Next, the data set 80 is trained on a neural network 85. As the neural network 85, it is preferable to use a neural network used for image generation, such as an autoencoder, a convolutional autoencoder (CAE), a variational autoencoder (VAE), a U-net, or a generative adversarial network (GAN).

[0085] Next, the generation of the target image in Fig. 8B will be described. Image 86 is an image of ingredient 62 acquired by the system. When image 86 is input to trained neural network 85, image 87 is obtained in which starting point 88 and end point 89 are added to ingredient 62. By displaying guideline 75 connecting starting point 88 and end point 89 through image processing, target image 90 in which knife guideline is added to the ingredient can be obtained.

[0086] Returning to FIG. 3 for further explanation, in step S7, the user selects whether to enable or disable the bone detection function. If bone detection is enabled, the system performs bone detection in step S9. The bone detection function is particularly effective in detecting small bones. By using the bone detection function, it is possible to prevent cooking or serving ingredients that contain small bones. This detection function is also effective in detecting scales, parasites, and other things in addition to bones. Bones, scales, parasites, and other things that are not suitable for consumption are referred to as foreign bodies. Therefore, the bone detection function can be called a foreign body detection function.

[0087] 9 shows what the field of view 50 looks like with bone detection enabled. Ingredient 76 represents a fish fillet. For example, ingredient 76 is a portion of a fillet cut from ingredient 62. When viewed through the AR glasses 10, bones 78 contained within ingredient 76 are highlighted, allowing the user to easily find and remove bones from the fish fillet regardless of their size.

[0088] Furthermore, text 73 may be displayed for the detected bone 78. The text 73 preferably includes information about the bone 78 and advice on how to deal with it. For example, if the detected bone 78 is a small bone that should be removed, information that a small bone has been detected and advice urging the removal of the bone may be displayed as text 73.

[0089] A method for detecting bones in a fish will be described with reference to Figures 10A and 10B. It is preferable to use a Semantic Segmentation method for detecting bones.

[0090] 10A shows a method for training a neural network using training data. The training is preferably performed by the server 23 shown in FIG.

[0091] Fig. 10B shows a method for generating a target image using a trained neural network. The process shown in Fig. 1B is performed inside the AR glasses 10. The AR glasses 10 download the trained neural network from the server 23 and generate a target image.

[0092] First, the learning method of Fig. 10A will be described. A data set 100 is prepared as training data. Data set 100 consists of image set 91 and image set 92. Image set 91 consists of images of multiple ingredients (in this case, fish fillets). In image set 92, each image in image set 91 is colored differently to indicate bone regions and other regions.

[0093] Next, the dataset 100 is trained on a neural network 95. As the neural network 95, a neural network used for semantic segmentation, such as FCN, SegNet, U-net, or PSPNet, is preferable.

[0094] Next, the generation of the target image in Fig. 10B will be described. Image 96 is an image of material 76 acquired by the system. When image 96 is input to trained neural network 95, image 97 is obtained in which material 76 is colored into bones and other areas. By combining image 97 and image 96, image 98 can be obtained in which the bones of material 76, especially small bones, are emphasized.

[0095] In addition, the above system can detect not only bones but also fish scales and parasites that infest fish by using technologies such as object detection and semantic segmentation. If the material 76 is not fish, hair and feathers are also included as foreign objects. If the material is a crustacean or shellfish, pieces of shells and shellfish, sand, etc. are also included as foreign objects. Hair, lint, fibers, and other parts of clothing of users and people involved in cooking such as chefs are also included as foreign objects. It is preferable that the detection function of the above system detects all foreign objects that are not suitable for consumption. In particular, it is unhygienic for hair to be attached to or mixed into the material. It is preferable to detect and remove it using the above system.

[0096] Finally, in step S9, the user selects "Exit" and the system is terminated (step S10).

[0097] As described above, one aspect of the present invention makes it possible to provide an information processing system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing method that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance method that allows information to be obtained without using hands.

[0098] The configurations, structures, methods, and the like described in this embodiment mode can be used in appropriate combination with the configurations, structures, methods, and the like described in other embodiment modes and examples.

[0099] (Embodiment 2) In the above embodiment, fish is used as an example of the ingredient, but the present invention is not limited to this. Other ingredients may be seafood other than fish, meat from mammals such as beef, pork, and lamb, meat from birds such as chicken, duck, and turkey, reptiles such as snakes and lizards, amphibians such as frogs, insects such as crickets, vegetables, fruits, and mushrooms. In the present embodiment, an example is shown in which vegetables are used as the ingredient 77.

[0100] Note that cooking utensils and items displayed on display unit 14 that are common to the previous embodiment are denoted by the same reference numerals, and detailed explanations may be omitted.

[0101] 11A shows the field of view 50 of a user looking at ingredients 77 placed on a cutting board 60 through the AR glasses 10. A recipe 71 and a guideline 75 are displayed on the display unit 14 of the AR glasses 10. The user can cut the ingredients 77 while aligning a knife with the guideline 75.

[0102] 11B shows the AR glasses 10 detecting the user's line of sight and selecting a desired step from the recipe 71. A pointer 51 is displayed on the recipe 71 in line with the user's line of sight, and the user can select a step while checking the position of the pointer 51.

[0103] When the user selects a desired step, an image of the selected step is displayed large. In addition, text 72 explaining the step and text 73 explaining useful advice are displayed below the field of view 50. This allows the user to understand important points for each step. Note that the images displayed in the recipe 71 are not limited to still images, and moving images may also be used.

[0104] A method for displaying the knife guideline in the system will be described with reference to Figures 12A and 12B. It is preferable that the knife guideline is displayed using machine learning using a neural network.

[0105] As with the previous embodiment, the system detects the position of the cutting board 60 using the first camera module 11 to detect the markers 61a-61d (FIG. 12A).

[0106] Next, ingredients 77 on the cutting board are detected, and candidates for ingredients 77 and candidates for recipes are displayed to the user. The user identifies ingredients 77 and selects a recipe using a method similar to that of the previous embodiment.

[0107] The system obtains information about the ingredients 77 and recipes for the ingredients 77, i.e., cooking methods and processing methods, from a database. FIG. 12B shows an example of cutting the ingredients 77 into small pieces (julienne). The system displays a guideline 75 on the display unit 14 so as to overlap with a desired position of the ingredients 77. For example, the system displays the guideline 75 at a position at a determined distance from the edge of the ingredients 77.

[0108] At this time, there is no limit to the number of displayed guidelines 75. One guideline 75 may be displayed for one cutting operation, or multiple guidelines 75 required for multiple cutting operations may be displayed.

[0109] As in the previous embodiment, the cutting positions for the material 77 can be learned by having a neural network learn training data. As the neural network, it is preferable to use a neural network used for image generation, such as Autoencoder, CAE, VAE, U-net, or GAN.

[0110] When the image of the material 77 acquired by the system is input to the trained neural network, information on the cutting position for the material 77 is obtained. Based on this information, image processing is performed and the guide line 75 is added to the material 77, thereby obtaining an image 99.

[0111] As described above, one aspect of the present invention makes it possible to provide an information processing system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing method that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance method that allows information to be obtained without using hands.

[0112] The configurations, structures, methods, and the like described in this embodiment mode can be used in appropriate combination with the configurations, structures, methods, and the like described in other embodiment modes and examples.

[0113] (Embodiment 3) In this embodiment, a method for displaying the processing time for a cooking utensil during cooking on the AR glasses 10 will be described.

[0114] When cooking ingredients and water using cooking utensils such as a pot, frying pan, or kettle, the heating time varies depending on the cooking contents, ingredients, and amounts of ingredients.

[0115] While performing the cooking and processing described in the above embodiment, the system detects the cooking utensils being used for cooking and displays the heating time for the cooking utensils and the time remaining until the cooking is completed on the AR glasses 10. By performing object detection using this system, even when cooking simultaneously using multiple cooking utensils, the heating time for each cooking utensil and the time remaining until the cooking is completed can be displayed on the AR glasses 10.

[0116] 13A shows a field of view 50 of a user watching through the AR glasses 10 how cooking utensils 210, 212, and 214 are arranged on a stove 200 having multiple heating means and are being heated and cooked by each heating means. The heating means of the stove 200 can use gas heating, induction heating (IH), heating by electric resistance, etc.

[0117] The heating means can adjust the heating intensity. In heating means that uses gas for heating, the heating intensity, i.e., the heat, can be adjusted by the amount of gas introduced. In heating means that uses induction heating or electrical resistance for heating, the heating intensity can be adjusted by the introduced power.

[0118] Fig. 13A shows an example in which ingredients that require 15 minutes of cooking are being heated using a cooking utensil 210. Fig. 13A shows a state in which 10 minutes have passed since the start of cooking. Also, a state in which text 220a indicating the elapsed time of cooking and the remaining heating time is displayed on the display unit 14 of the AR glasses 10 is shown.

[0119] 13A shows an example in which ingredients that require cooking for 10 minutes are being heated using a cooking utensil 212. FIG. 13A shows a state in which 2 minutes have passed since the start of cooking. Also, the display unit 14 of the AR glasses 10 shows a state in which text 220b indicating the elapsed time of cooking and the remaining heating time is displayed.

[0120] 13A also shows an example in which water is heated to 100°C using cooking utensil 214. FIG 13A also shows the state in which the water in cooking utensil 214 has been heated to 80°C.

[0121] The cooking utensil 214 is also provided with a temperature sensor 216 that can wirelessly communicate with the AR glasses 10. The temperature inside the cooking utensil 214 detected by the temperature sensor 216 is transmitted to the AR glasses 10 and can be displayed on the display unit 14.

[0122] The system estimates the heating time required to reach the desired temperature based on the amount of water in the cooker 214, the temperature of the water before heating, and the change in temperature of the water due to heating. The heating time estimation is preferably performed in a database. Alternatively, the heating time estimation may use machine learning or a data sheet indicating the energy required to heat a certain amount of water.

[0123] FIG. 13A shows how the current temperature of the water in the cooking utensil 214 and a prediction that it will reach 100° C. in two minutes are displayed in text 220c.

[0124] 13A, the cooking utensils 210 and 212 may also be provided with temperature sensors 216, similar to the cooking utensil 214. When cooking with oil, such as frying or deep-frying, using the cooking utensils 210 and 212, the temperature sensor 216 can obtain the temperature of the oil and transmit it to the AR glasses 10. The AR glasses 10 can display the internal temperature of each cooking utensil on the display unit 14.

[0125] The user can cook at an optimal temperature while checking the temperature inside the cooking utensil. Moreover, excessive heating of the oil inside the cooking utensil may cause the oil to heat up and ignite. When the temperature of the oil exceeds a certain value, the AR glasses 10 can display a warning on the display unit 14 to alert the user. Moreover, the AR glasses 10 may be connected to the stove 200 via a network. In this case, it is preferable to stop heating of the desired heating unit by a signal emitted from the AR glasses 10.

[0126] Furthermore, the cooking recipe and temperature information inside the cooking utensil obtained from the temperature sensor 216 may be analyzed using a database, and the time required for cooking may be calculated and displayed in the text 220a, 220b, and 220c.

[0127] FIG. 13B shows the field of view 50 of the user looking through the AR glasses 10 at the scene on the stove 200 five minutes after the state of FIG. 13A.

[0128] 13B shows the end of a predetermined heating time during cooking using the cooking utensil 210. The text 220a indicates that the heating time has ended. When the predetermined heating time has elapsed, the AR glasses 10 may transmit a signal to the stove 200, and the stove 200 may stop heating the cooking utensil 210 upon receiving the signal.

[0129] In the case of cooking using the cooking utensil 212, FIG. 13B shows that 7 minutes have passed since the start of cooking. Also, the text 220b on the display unit 14 of the AR glasses 10 shows that the elapsed time of cooking and the remaining heating time are being updated. At this time, the AR glasses 10 may analyze the temperature obtained from the temperature sensor 216 provided in the cooking utensil 212 and transmit a signal to the stove 200 to adjust the heating intensity (heat power, etc.) of the cooking utensil 212. For example, if the inside of the cooking utensil 212 is overheated, the AR glasses 10 can transmit a signal to the stove 200 to reduce the heating. Also, if the internal temperature of the cooking utensil 212 is low, the AR glasses 10 can transmit a signal to the stove 200 to increase the heating. The stove 200 that receives the signal adjusts the heating intensity of the corresponding heating unit.

[0130] In Figure 13B, cooking utensil 214 has already been used to heat water. Figure 13B shows that the user has removed cooking utensil 214 from stove 200. Because cooking utensil 214 is no longer in field of view 50, text 220c is no longer displayed on display 14.

[0131] The system may end the heating when the water inside the cooking utensil 214 reaches a desired temperature. The system may also end the heating when it detects that the cooking utensil 214 is moved away from the heating means by image processing, which will be described later. The end of the heating may be achieved by the AR glasses 10 sending a signal to the stove 200 to end the heating of the desired heating means when the stove 200 receives the signal.

[0132] Next, a method for detecting cooking utensils such as cooking utensil 210, cooking utensil 212, and cooking utensil 214 will be described with reference to FIG. 14A and FIG. 14B. For detecting cooking utensils, object detection methods such as R-CNN, YOLO, and SSD, and semantic segmentation methods such as FCN, SegNet, U-Net, and PSPNet, which were described in the background art, can be used. The system can identify the type of cooking utensil by analyzing the image of the shape, size, and other features of the cooking utensil. Shape features include the shape and number of handles that the cooking utensil has, and the shape of the spout. Size features include the area of ​​the base, the height, and the volume.

[0133] 14A shows that rectangle 222a surrounding cooking utensil 210, rectangle 222b surrounding cooking utensil 212, and rectangle 222c surrounding cooking utensil 214 are displayed on display unit 14. The system performs image analysis of the inside of rectangle 222a to determine that cooking utensil 210 is a pot, and displays text 224a. The system also performs image analysis of the inside of rectangle 222b to determine that cooking utensil 212 is a frying pan, and displays text 224b. The system also performs image analysis of the inside of rectangle 222c to determine that cooking utensil 214 is a kettle, and displays text 224c.

[0134] The type of each cooking utensil may be determined by machine learning using training data, or the user may register the cooking utensils they will use in advance in a database or in the memory 16 of the AR glasses 10, and the determination may be based on the registered data.

[0135] Alternatively, each cooking utensil may be divided into regions as shown in Fig. 14B. The type of cooking utensil may be identified from the shape of each region and a label may be assigned to each region, or the type of cooking utensil may be identified by the image analysis described above and then a label may be assigned to each region.

[0136] When a cooking utensil is detected within the field of view 50 by the above method, the system displays information about the cooking utensil on the display unit 14. Information displayed on the display unit 14 may include the type of cooking utensil, the ingredients contained in the cooking utensil, cooking time, etc. Furthermore, if a cooking utensil is no longer present in the field of view 50 due to a movement of the user's line of sight or the movement of the cooking utensil, the information about the cooking utensil may be hidden. Alternatively, the information about the cooking utensil may be hidden after a certain period of time has passed since no cooking utensil was present in the field of view 50.

[0137] As described above, one aspect of the present invention makes it possible to provide an information processing system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide an information processing method that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance system that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance device that allows information to be obtained without using hands. Another aspect of the present invention makes it possible to provide a cooking assistance method that allows information to be obtained without using hands.

[0138] The configurations, structures, methods, and the like described in this embodiment mode can be used in appropriate combination with the configurations, structures, methods, and the like described in other embodiment modes and examples.

[0139] (Embodiment 4) In this embodiment, a configuration example of an information terminal equipped with a display device that can be used for the AR glasses 10 will be described.

[0140] 15A shows a perspective view of a glasses-type information terminal 900. The information terminal 900 includes a pair of display panels 901, a pair of housings (a housing 902a and a housing 902b), a pair of optical members 903, a pair of mounting units 904, and the like.

[0141] The information terminal 900 can project an image displayed on the display panel 901 onto a display area 906 of the optical member 903. Since the optical member 903 has translucency, the user can see the image displayed on the display area 906 superimposed on the transmitted image visually recognized through the optical member 903. Therefore, the information terminal 900 is an information terminal capable of AR display. The display unit 14 described in the above embodiment can include not only the display panel 901 but also an optical member 903 including the display area 906, and an optical system having a lens 911, a reflector 912, and a reflecting surface 913 described later. The display panel 901 can be an organic EL display, an LED display, an inorganic EL display, a liquid crystal display, or the like. When a liquid crystal display is used as the display panel 901, it is preferable to provide a light source that functions as a backlight.

[0142] The information terminal 900 is provided with a pair of cameras 905 capable of capturing an image of the front and a pair of cameras 909 capable of capturing an image of the user side. The cameras 905 are part of the components of the first camera module 11, and the cameras 909 are part of the components of the second camera module 12. By providing a plurality of cameras 905 in the information terminal 900, it is possible to capture images of ingredients and cooking utensils in a three-dimensional manner, which is preferable. However, the camera 905 in this embodiment is not limited to this. The number of cameras 905 provided in the information terminal 900 may be one. In this case, the camera 905 may be provided in the center of the front surface of the information terminal 900, or may be provided on the front surface of one of the housings 902a and 902b. In addition, two cameras 905 may be provided on the front surfaces of the housings 902a and 902b, respectively.

[0143] The camera 909 can detect the line of sight of the user. Therefore, it is preferable to provide two cameras 909, one for the right eye and one for the left eye. However, if one camera can detect the line of sight of both eyes, only one camera 909 may be used. The camera 909 may also be an infrared camera that can detect infrared rays.

[0144] The housing 902a includes a wireless communication device 907, which can supply a video signal or the like to the housing 902. The wireless communication device 907 preferably includes a communication module 17 and communicates with a database. Instead of the wireless communication device 907, or in addition to the wireless communication device 907, a connector may be provided to which a cable 910 to which a video signal or a power supply potential is supplied can be connected. The cable 910 may function as a wiring 10c that connects to the housing 10b. By providing an acceleration sensor, a gyro sensor, or the like in the housing 902 as the sensor 25, the direction of the user's head can be detected and an image corresponding to the direction can be displayed in the display area 906. The housing 902 preferably includes a battery 21, which can be charged wirelessly or by wire.

[0145] The housing 902b is also provided with an integrated circuit 908. The integrated circuit 908 has a controller 13, a processor 15, a memory 16, an audio controller 18, etc., and has a function of controlling various components of the information terminal 900, such as a camera 905, a wireless communication device 907, a pair of display panels 901, a microphone 19, a speaker 20, etc., a function of generating an image, etc. The integrated circuit 908 may have a function of generating a composite image for AR display.

[0146] Data communication with external devices can be performed by the wireless communication device 907. For example, data transmitted from the outside can be output to the integrated circuit 908, and the integrated circuit 908 can generate image data for AR display based on the data. Examples of data transmitted from the outside include data including information necessary for cooking transmitted from a database, as well as data including information related to cooking transmitted from various sensors provided in the cooking utensil.

[0147] 15B, a method of projecting an image onto the display area 906 of the information terminal 900 will be described. A display panel 901, a lens 911, and a reflector 912 are provided inside the housing 902. In addition, a portion of the optical member 903 that corresponds to the display area 906 has a reflecting surface 913 that functions as a half mirror.

[0148] Light 915 emitted from the display panel 901 passes through the lens 911 and is reflected by the reflector 912 toward the optical member 903. Inside the optical member 903, the light 915 is repeatedly totally reflected at the end surface of the optical member 903 and reaches the reflecting surface 913, whereby an image is projected onto the reflecting surface 913. This allows the user to visually recognize both the light 915 reflected by the reflecting surface 913 and the transmitted light 916 transmitted through the optical member 903 (including the reflecting surface 913). When a liquid crystal display is used as the display panel 901, it is preferable that a light source functioning as a backlight is provided so that light from the light source passes through the display panel 901 and enters the lens 911. That is, it is preferable that the liquid crystal panel of the liquid crystal display is provided between the light source and the lens 911.

[0149] 15B shows an example in which the reflector 912 and the reflecting surface 913 each have a curved surface. This allows for greater freedom in optical design compared to when these surfaces are flat, and allows for a thinner optical member 903. The reflector 912 and the reflecting surface 913 may also be flat.

[0150] A member having a mirror surface can be used as the reflector 912, and it is preferable that the reflector has a high reflectance. Also, a half mirror utilizing the reflection of a metal film may be used as the reflector surface 913, but the transmittance of the transmitted light 916 can be increased by using a prism utilizing total reflection or the like.

[0151] Here, the housing 902 preferably has a mechanism for adjusting the distance between the lens 911 and the display panel 901 and the angle between them. This makes it possible to adjust the focus and enlarge or reduce the image. For example, the lens 911 or the display panel 901 or both may be configured to be movable in the direction of the optical axis.

[0152] Furthermore, it is preferable that the housing 902 has a mechanism capable of adjusting the angle of the reflector 912. It is possible to change the position of the display area 906 where an image is displayed by changing the angle of the reflector 912. This makes it possible to position the display area 906 at an optimal position according to the position of the user's eyes.

[0153] The display device of one embodiment of the present invention can be applied to the display panel 901. Thus, the information terminal 900 can display images with extremely high definition. [Explanation of symbols]

[0154] 10: AR glasses, 10a: glass unit, 10b: housing, 10c: wiring, 11: camera module, 12: camera module, 13: controller, 14: display unit, 15: processor, 16: memory, 17: communication module, 18: audio controller, 19: microphone, 20: speaker, 21: battery, 22: bus, 23: server, 25: sensor, 26: computer, 27: computer, 28: computer, 29: smartphone, 50: field of view, 51: pointer, 60: cutting board, 61a: marker, 61b: marker, 61c: marker, 61d: marker, 62: ingredients, 63: knife, 64: dishcloth, 65: rectangle, 70: menu, 71: recipe, 72: text, 73: text, 75: guideline, 76: ingredients, 77: ingredients, 78: bone, 80: dataset, 81: image set, 82: image set, 83: start point, 84: end point, 85: neural network, 86: image, 87: image, 88: start point, 89: end point, 90: image, 91: image set, 92: image set, 95: neural network, 96: image, 97: image, 98: image, 99: image, 100: dataset, 200: stove, 210: cookware, 212: cookware, 214: cookware, 216: temperature sensor, 220a: text, 220b: text, 220c: text, 222a: rectangle, 2 22b: rectangle, 222c: rectangle, 224a: text, 224b: text, 224c: text, 900: information terminal, 901: display panel, 902: housing, 902a: housing, 902b: housing, 903: optical member, 904: mounting part, 905: camera, 906: display area, 907: wireless communication device, 908: integrated circuit, 909: camera, 910: cable, 911: lens, 912: reflector, 913: reflecting surface, 915: light, 916: transmitted light

Claims

1. A wearable device including a display means and an imaging means; a cooking utensil having a temperature sensor; the wearable device and the temperature sensor are connected via a first network; The wearable device has a function of detecting the cooking utensil by the imaging means, The wearable device has a function of collecting information on a temperature inside the cooking utensil from the temperature sensor, When the cooking utensil is present in a specific area captured by the imaging means, the display means displays information regarding the temperature; When the cooking utensil is not present in the area, the information processing system hides the information regarding the temperature from the display means.

2. In claim 1, The information processing system further includes a database, the database is connected to the wearable device and the temperature sensor via a second network including the first network; the database receives information regarding the temperature via the second network; The database calculates the time required to heat the cooking utensil from the information relating to the temperature, and displays the time on the display means.

3. 3. The information processing system according to claim 1, wherein the wearable device is a glasses-type wearable device.

4. 3. An information processing system according to claim 2, wherein the database is stored in a server.

Images