Menu estimation device, menu estimation method, and program

JP7595438B2Active Publication Date: 2024-12-06NTT COMWARE CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020181561
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-29
Publication Date
2024-12-06
Estimated Expiration
2040-10-29

AI Technical Summary

Benefits of technology

【0019】 本発明の一態様によれば、メニューの推定精度を高くすることができる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595438000005
    Figure 0007595438000005
  • Figure 0007595438000006
    Figure 0007595438000006
  • Figure 0007595438000007
    Figure 0007595438000007
Patent Text Reader

Abstract

To provide a menu estimation apparatus and method for improving the accuracy of estimating a menu, a volume estimation apparatus and method, and a program.SOLUTION: A menu estimation system includes: a model learning unit 300 which trains an estimation model to output a menu of food items included in an acquired image; an estimation unit 400 which inputs the acquired image, as a target image, to the estimation model and estimates a menu of the food items included in the target image on the basis of an output of the estimation model; and an intermediate output unit 500 which outputs first menu information indicating the menu estimated by the estimation unit 400 and first area information indicating an image area in the target image corresponding to the first menu information.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a menu estimation device and method, a volume estimation device and method, and a program. [Background technology]

[0002] Conventionally, there have been known techniques for estimating an object in an image. For example, an object recognition system described in Patent Document 1 is known as such a technique. This object recognition system acquires image data of an object and image data of an object before and after manipulation of an acted object that is affected by the manipulation of the object, extracts shape features of the object from the image data of the object, learns the object based on the shape of the object recognized using the shape features in an object concept model, and applies observed information on the shape of the object to be recognized to the object concept model of the object to recognize the object to be recognized. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-123365 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a technology that uses an image as input information and estimates objects contained in the image, such as the object recognition system described above, food menus in an image are estimated. However, when the accuracy of the menu estimation is low, it is unclear which part of the input image was focused on to estimate the size (volume) of the food. In addition, in a conventional technology, a technology is considered in which the size of the food is estimated by comparing it with a reference object such as chopsticks or a spoon in the image, the size of which is known in advance, but the estimation accuracy may decrease if the reference object does not exist. In addition, the accuracy of identifying the menu and estimating the volume may decrease depending on the environmental difference such as the angle at which the input image was captured.

[0005] The present invention has been made in consideration of the above problems, and aims to provide a menu estimation device and method, a volume estimation device and method, and a program that can improve the estimation accuracy of a menu. [Means for solving the problem]

[0006] (1) One aspect of the present invention includes an image acquisition unit that acquires an image, and a learning unit that receives the image acquired by the image acquisition unit as learning data, trains a menu estimation model to output information about a menu of a food included in the image acquired by the image acquisition unit, and trains a volume estimation model to output information about the volume of the menu, the information about the volume including information about a distance from an imaging device that captured the image acquired by the image acquisition unit to the food, a viewpoint of the imaging device with respect to the food, incident light and reflected light with respect to the food, and environmental information about lighting, inputting the image acquired by the image acquisition unit as a target image into the menu estimation model, estimating information about the menu of the food included in the target image based on an output of the menu estimation model, and outputting first menu information and a reliability level related to the menu estimated by the menu estimation model, inputting the image acquired by the image acquisition unit as a target image into the volume estimation model, estimating information about the volume of the menu based on the output of the volume estimation model, and learning the volume estimation model an estimation unit that outputs first volume information relating to the volume of a menu estimated by the estimation unit, and a reliability determined by a function that inputs information on a viewpoint of the imaging device, incident light and reflected light on food, and lighting; a first output unit that outputs the first menu information and the first volume information estimated by the estimation unit, and first region information indicating an image region in the target image corresponding to the first menu information and the first volume information; a correction unit that receives an instruction to correct the first menu information, the first volume information, and the first region information output from the first output unit, and corrects the menu and the reliability corresponding to the menu, the image region, the distance information, and the environmental information based on the received instruction; a second output unit that outputs second menu information and the reliability corresponding to the menu that are corrected by the correction unit, and second region information indicating the image region, the distance information, and the environmental information; The original imagea learning data generation unit that determines whether to generate a learning image using the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu, and that generates an image as learning data, the learning unit learning the menu estimation model using the image generated by the learning data generation unit, the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu; Using the first volume information, the generated image as learning data, the first region information, the second region information, the corrected distance information, the corrected environment information, and a reliability corresponding to the volume, The menu estimation device is configured to train the volume estimation model.

[0008] ( 2 2) One aspect of the present invention is the menu estimation device described above, wherein the estimation unit estimates a menu included in the target image and a part included in the menu, The first output section may output character information indicating the menu and character information indicating a portion included in the menu.

[0009] ( 3 ) One aspect of the present invention is the above-mentioned menu estimation device, wherein when the estimation unit estimates multiple menus for one image area included in the target image, the estimation unit may output information indicating the image area, text information indicating each menu and the reliability of each menu, and text information indicating that a selection of one of the multiple menus is being prompted.

[0010] ( 4 ) One aspect of the present invention is the above-mentioned menu estimation device, wherein, when a menu cannot be estimated for an image area included in the target image, the estimation unit may output information indicating the image area and text information prompting the user to input a menu.

[0013] ( 5 2) One aspect of the present invention is the menu estimation device described above, further comprising a nutrient estimation unit that estimates nutrients of the menu based on information related to the food menu and the volume of the menu.

[0015] ( 6) One aspect of the present invention includes the steps of acquiring an image, inputting the acquired image as learning data, training a menu estimation model to output information about a menu of a food included in the acquired image, and training a volume estimation model to output information about the volume of the menu, the information about the volume including distance information from an imaging device that captured the acquired image to the food, a viewpoint of the imaging device with respect to the food, incident light and reflected light on the food, and environmental information related to lighting, inputting the acquired image as a target image to the menu estimation model, estimating information about the menu of the food included in the target image based on an output of the menu estimation model, and outputting first menu information and a reliability regarding the menu estimated by the menu estimation model, inputting the acquired image as a target image to the volume estimation model, estimating information about the volume of the menu based on the output of the volume estimation model, and outputting the first volume information regarding the volume of the menu estimated by the volume estimation model, and the viewpoint of the imaging device, the incident light on the food, and environmental information related to lighting. outputting a reliability determined by a function to which information on light, reflected light, and illumination is input; outputting the estimated first menu information and first volume information, and first region information indicating an image region in the target image corresponding to the first menu information and the first volume information; receiving an instruction to correct the outputted first menu information, the first volume information, and the first region information, and correcting the menu and the reliability corresponding to the menu, the image region, the distance information, and the environmental information based on the received instruction; outputting second menu information indicating information related to the corrected menu, the reliability corresponding to the menu, and second region information indicating the image region, the distance information, and the environmental information; and determining whether or not to generate a learning image using the original image based on the reliability corresponding to the menu and the reliability corresponding to the volume attached to the original image, and generating an image as learning data, wherein the step of training an estimation model includes:training the menu estimation model using the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu; Using the first volume information, the generated image as learning data, the first region information, the second region information, the corrected distance information, the corrected environment information, and a reliability corresponding to the volume, The menu estimation method further comprises training the volume estimation model.

[0016] ( 7) One aspect of the present invention includes a method for training a computer to acquire an image, input the acquired image as learning data, train a menu estimation model to output information about a menu of food included in the acquired image, and train a volume estimation model to output information about the volume of the menu, the information about the volume including information about a distance from an imaging device that captured the acquired image to the food, a viewpoint of the imaging device with respect to the food, incident light and reflected light on the food, and environmental information about lighting, inputting the acquired image as a target image to the menu estimation model, estimating information about the menu of food included in the target image based on an output of the menu estimation model, and outputting first menu information and a reliability level about the menu estimated by the menu estimation model, inputting the acquired image as a target image to the volume estimation model, and a step of estimating information regarding the volume of the menu based on an output of an estimation model, and outputting first volume information regarding the volume of the menu estimated by the volume estimation model, and a reliability determined by a function to which information regarding the viewpoint of the imaging device, the incident light and reflected light on the food, and the lighting is input; a step of outputting the estimated first menu information and the first volume information, and first region information indicating an image region in the target image corresponding to the first menu information and the first volume information; a step of receiving an instruction to correct the outputted first menu information, the first volume information, and the first region information, and correcting the menu, the reliability corresponding to the menu, the image region, the distance information, and the environmental information based on the received instruction; and a step of outputting second menu information indicating the corrected information regarding the menu, the reliability corresponding to the menu, and the second region information indicating the image region, the distance information, and the environmental information.determining whether or not to generate a learning image using the original image based on the reliability corresponding to the menu and the reliability corresponding to the volume associated with the original image, and generating an image as learning data; and the step of training the estimation model includes training the menu estimation model using the image as the generated learning data, the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu; Using the first volume information, the generated image as learning data, the first region information, the second region information, the corrected distance information, the corrected environment information, and a reliability corresponding to the volume, A program for training the volume estimation model. Effect of the Invention

[0019] According to one aspect of the present invention, it is possible to improve the accuracy of menu estimation. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram illustrating an example of a functional configuration of a menu estimation system according to an embodiment. [Diagram 2] FIG. 11 is a block diagram showing an example of a learning process of a menu estimation model in the embodiment. [Diagram 3] FIG. 11 is a block diagram showing an example of an estimation process of a menu estimation model in the embodiment. [Figure 4] FIG. 11 is a block diagram showing an example of a learning process of a volume estimation model in an embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of an estimation process of a volume estimation model in an embodiment. [Figure 6] FIG. 11 is a diagram showing an example of information presented to a user in the embodiment. [Figure 7] FIG. 13 is a diagram illustrating a feature map in the embodiment. [Figure 8] FIG. 13 is a diagram showing an image and areas of a food product captured from above. [Figure 9] 1 is an image showing an image of a food product taken from the side and an image showing a region. [Figure 10]FIG. 2 is a block diagram showing an example of a nutrient estimation unit in the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a data generation process according to an embodiment. [Figure 12] 10A to 10C are diagrams illustrating another example of the data generation process in the embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of an estimation result in the embodiment. [Figure 14] FIG. 11 is a diagram illustrating an operation of a learning data generating unit according to the embodiment. [Figure 15] 10A to 10C are diagrams illustrating an example of a generation process of a learning image in an embodiment. [Figure 16] 11A to 11C are diagrams illustrating another example of the process of generating learning images in the embodiment. [Figure 17] 11A to 11C are diagrams illustrating another example of the process of generating learning images in the embodiment. [Figure 18] 11A to 11C are diagrams illustrating another example of the process of generating learning images in the embodiment. [Figure 19] FIG. 13 is a diagram showing a specific example of adjusting a set of learning images. [Figure 20] FIG. 13 is a diagram showing a specific example of adjusting a set of learning images. [Figure 21] FIG. 13 is a block diagram showing an evaluation of an estimation model in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A menu estimation device and method, a volume estimation device and method, and a program to which the present invention is applied will be described below with reference to the drawings.

[0022] <Overview of the embodiment> The menu estimation system 1 of the embodiment collects images, trains a menu estimation model using the collected images, and estimates the menu of food included in the target image using the menu estimation model. The menu estimation system 1 also collects images, trains a volume estimation model using the collected images, and estimates the volume of food included in the target image using the volume estimation model. In this way, when the menu estimation system 1 receives a target image and a request, for example, it can realize a service that provides information based on the menu and volume of food included in the target image. The menu estimation system 1 will be described in detail below.

[0023] <Configuration of Menu Estimation System 1> FIG. 1 is a block diagram showing an example of a functional configuration of a menu estimation system 1 according to an embodiment. The menu estimation system 1 includes, for example, a user terminal device 100, an image storage unit 200, a learning data generation unit 210, a model learning unit 300, an estimation unit 400, an intermediate output unit (first output unit) 500, a correction unit 600, and an output unit (second output unit) 700. The user terminal device 100, the image storage unit 200, the model learning unit 300, the estimation unit 400, the intermediate output unit 500, the correction unit 600, and the output unit 700 are connected to, for example, a communication network. Each device connected to the communication network includes a communication interface such as a network interface card (NIC) or a wireless communication module (not shown in FIG. 2). The communication network includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a cellular network, and the like.

[0024] The user terminal device 100 is, for example, a terminal device such as a personal computer, a smartphone, a tablet terminal, etc. The user terminal device 100, for example, accepts an operation by an administrator of the menu estimation system 1, displays an estimation result, and generates information for correcting the estimation result.

[0025] The image accumulation unit 200 includes, for example, an information processing device that collects images, and a storage device. The image accumulation unit 200 collects images and accumulates the collected images. The storage is realized by a storage device such as a HDD, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The log data accumulation unit 206 and the detection result accumulation unit 208 may be realized by, for example, a SAN (Storage Area Network) or a NAS (Network Attached Storage).

[0026] The learning data generation unit 210 is realized by a processor such as a CPU (Central Processing Unit) executing a program stored in a program memory. The program may be stored in advance in a storage device (a storage device equipped with a non-transitory storage medium) such as a HDD or flash memory. The learning data generation unit 210 generates learning data used for model learning using images stored in the image storage unit 200, and stores the data in the image storage unit 200.

[0027] The model learning unit 300 includes a learning processing unit 310. The learning processing unit 310 is realized by a processor such as a CPU executing a program stored in a program memory. The learning processing unit 310 executes a learning process to construct a menu estimation model and a volume estimation model. The estimation model in the embodiment is, for example, a neural network, but is not limited thereto, and may be anything other than a neural network as long as it is a machine learning model that learns labeled images and outputs a label and a reliability for an input of a target image.

[0028] The estimation unit 400 includes a menu estimation unit 410 and a volume estimation unit 420. The menu estimation unit 410 and the volume estimation unit 420 are realized by a processor such as a CPU executing a program stored in a program memory. The menu estimation unit 410 executes processing using a menu estimation model 412. The menu estimation model 412 is a machine learning model that receives a target image as an input and outputs a food menu as an estimation result. The volume estimation unit 420 executes processing using a volume estimation model 422. The volume estimation model 422 is a machine learning model that receives a target image as an input and outputs information about the volume of the menu. The target image is an image for which the menu and / or volume are to be estimated. In the embodiment, the menu estimation model 412 and the volume estimation model 422 are separate, but are not limited to this, and may be one machine learning model. That is, instead of the menu estimation model 412 and the volume estimation model 422, one machine learning model may be provided that receives a target image as an input and outputs information about the food menu and the volume of the menu.

[0029] The intermediate output unit 500 is, for example, an interface circuit communicatively connected to the user terminal device 100 and the image storage unit 200. The intermediate output unit 500 receives menu information 510 as an estimation result from the menu estimation unit 410, and supplies the menu information 510 to the correction unit 600. The intermediate output unit 500 also outputs information based on the menu information 510 to the user terminal device 100 and the image storage unit 200. The menu information 510 includes, for example, a menu ID, a reliability, and area information. The menu ID is information for identifying a menu (first menu information). The area information is information indicating an area occupied by a menu included in an image (first area information). The reliability is information indicating the likelihood of the estimation result (menu ID and area information).

[0030] The intermediate output unit 500 inputs volume information 520 as an estimation result from the volume estimation unit 420, and supplies the volume information 520 to the correction unit 600. The intermediate output unit 500 also outputs information based on the volume information 520 to the user terminal device 100 and the image storage unit 200. The volume information 520 includes, for example, area information, distance information, and environmental information. The area information, distance information, and environmental information are examples of information related to the volume of a menu. The area information is information indicating an area occupied by a menu included in an image. The distance information is information indicating the distance from the imaging device that captured the image to the menu. The environmental information is information related to the viewpoint of the imaging device with respect to the menu, the incident light and reflected light with respect to the menu, and lighting.

[0031] The correction unit 600 includes a menu correction unit 610 and a volume correction unit 620. The menu correction unit 610 and the volume correction unit 620 are realized by a processor such as a CPU executing a program stored in a program memory. The menu correction unit 610 receives a request to correct a menu from the user terminal device 100, and corrects the menu ID, reliability, and area information according to the received request. The volume correction unit 620 receives a request to correct information related to the volume from the user terminal device 100, and corrects the area information, distance information, and environmental information according to the received request. The correction unit 600 outputs to the output unit 700 the menu information including the corrected menu ID# (second menu information), reliability#, and area information# (second area information), and the volume information including the corrected area information#, distance information#, and environmental information#.

[0032] The output unit 700 is, for example, an interface circuit communicatively connected to the user terminal device 100 and the image storage unit 200. The output unit 700 inputs the corrected menu information from the correction unit 600 and outputs the corrected menu information to the user terminal device 100 and the image storage unit 200. The output unit 700 inputs the corrected volume information from the correction unit 600 and outputs the corrected volume information to the user terminal device 100 and the image storage unit 200.

[0033] The intermediate output unit 500 inputs the volume information 520 as an estimation result from the volume estimation unit 420, and supplies the volume information 520 to the correction unit 600. The intermediate output unit 500 also outputs information based on the volume information 520 to the user terminal device 100 and the image storage unit 200. The volume information 520 includes, for example, area information, distance information, and environmental information. The area information, distance information, and environmental information are examples of information related to the volume of the menu. The area information is information indicating an area occupied by the menu included in the image. The distance information is information indicating a distance from the imaging device that captured the image to the menu. The environmental information is information related to the viewpoint of the imaging device with respect to the menu, the incident light and reflected light with respect to the menu, and the lighting. This allows the menu estimation system 1 to learn the menu estimation model 412 using the menu information before and after the correction. The menu estimation system 1 can also learn the volume estimation model 422 using the volume information before and after the correction.

[0034] FIG. 2 is a block diagram showing an example of a learning process of the menu estimation model in the embodiment. The learning processing unit 310 inputs learning data including image data, regions, labels, and reliability, and supplies it to the menu estimation model 412. The learning data is the menu information 510 output from the intermediate output unit 500, the corrected menu information output from the output unit 700, or the menu information acquired by the image storage unit 200 from an external device. The learning data may be either a correct image or a false image, or may contain both images. The label is information corresponding to the above-mentioned menu ID. The reliability is the likelihood that the menu corresponding to the label is included in the image data, and is a high value when, for example, the menu is clearly included in the image. The label and reliability may be automatically added to the image data, or may be added based on the operation of the user terminal device 100. The label and reliability are set corresponding to the region of the image data. When multiple regions exist for the image data, the label and reliability are set corresponding to each region.

[0035] The learning processing unit 310 supplies the learning data to the menu estimation model 412 (Process 1). The menu estimation model 412 outputs the estimation result for the learning data to the learning processing unit 310 (Process 2). When the learning data is a correct image, the learning processing unit 310 updates the processing parameters of the menu estimation model 412 so that the menu ID and area estimated by the menu estimation model 412 become the menu ID and area corresponding to the label in the learning data (Process 3). When the learning data is a misrecognized image, the learning processing unit 310 updates the processing parameters of the menu estimation model 412 so that the menu ID and area estimated by the menu estimation model 412 do not become the menu ID and area corresponding to the label in the learning data (Process 3). The processing parameters of the menu estimation model 412 are, for example, filters (also called weights or biases) included in a neural network.

[0036] 3 is a block diagram showing an example of an estimation process of a menu estimation model in an embodiment. The learned menu estimation model 412 is introduced into a menu estimation unit 410. When a target image is input, the menu estimation model 412 outputs, for each region, information indicating the region, a label, and a reliability as an estimation result.

[0037] FIG. 4 is a block diagram showing an example of a learning process of a volume estimation model in an embodiment. The learning processing unit 310 inputs learning data including image data, regions, labels, distance information, environmental information, and reliability, and supplies the learning data to the volume estimation model 422. The learning data is the volume information 520 output from the intermediate output unit 500, the corrected volume information output from the output unit 700, or the volume information acquired by the image storage unit 200 from an external device. The learning data may be either a correct image or a false image, or may contain both images. The label is information corresponding to the above-mentioned menu portion. The reliability is the likelihood that the portion of the menu corresponding to the label is included in the image data, and is a high value when, for example, the portion of the menu is clearly included in the image. The label and reliability may be automatically added to the image data, or may be added based on the operation of the user terminal device 100. The label, distance information, environmental information, and reliability are set corresponding to the region of the image data. When multiple regions exist for the image data, the label, distance information, environmental information, and reliability are set corresponding to each region.

[0038] The learning processing unit 310 supplies the learning data to the volume estimation model 422 (Process 1). The volume estimation model 422 outputs the estimation result for the learning data to the learning processing unit 310 (Process 2). When the learning data is a correct image, the learning processing unit 310 updates the processing parameters of the volume estimation model 422 so that the area, label, distance information, and environment information estimated by the volume estimation model 422 become the area, label, distance information, and environment information in the learning data (Process 3). When the learning data is a misrecognized image, the learning processing unit 310 updates the processing parameters of the volume estimation model 422 so that the area, label, distance information, and environment information estimated by the volume estimation model 422 do not become the area, label, distance information, and environment information in the learning data (Process 3). The processing parameters of the volume estimation model 422 are, for example, filters (also called weights and biases) included in a neural network.

[0039] 5 is a block diagram showing an example of an estimation process of a volume estimation model in an embodiment. The trained volume estimation model 422 is introduced into the volume estimation unit 420. When a target image is input, the volume estimation model 422 outputs, for each region, information indicating the region, distance, environment, and reliability as an estimation result.

[0040] 6 is a diagram showing an example of information presented to a user in an embodiment. The menu estimation system 1 outputs the menu ID estimated by the estimation unit 400 and area information indicating an area in a target image corresponding to the menu ID to the user terminal device 100, and the user terminal device 100 performs display using the target image, the menu ID, and the area information.

[0041] The estimation unit 400 may estimate a menu included in the target image and a portion included in the menu, and output character information indicating the menu and character information indicating the portion included in the menu. As a result, the user terminal device 100 can display character information indicating the result of estimating "ramen" as a menu, such as "Ramen A because it has a lot of bean sprouts on it," together with the target image, as shown in FIG. 6(a), and character information indicating the result of estimating the portion included in the menu, such as "it has a lot of bean sprouts on it," as the basis for the estimation. As a result, the user terminal device 100 can accept an input of whether the estimation result is correct or not based on the target image and the character information. For example, when the correction unit 600 accepts that the estimation result is correct, it can set the reliability of the label for the target image (possible values: 0 to 1) to "1."

[0042] When estimating a plurality of menus for one image region included in the target image, the estimation unit 400 may output information indicating the image region, text information indicating each menu and its reliability, and text information prompting the user to select one of the plurality of menus, as shown in Fig. 6(b). As a result, the user terminal device 100 can accept an operation to select an egg dish that is the correct answer based on the target image and the text information. For example, when the correction unit 600 accepts that "egg dish A" is the correct answer, it can change the reliability corresponding to the label of egg dish A included in the region of the target image to a higher one and change the reliability corresponding to the label of egg dish B to a lower one.

[0043] When the estimation unit 400 cannot estimate a menu for an image area included in the target image, it may output information indicating the image area and text information prompting input of a menu, such as "It is unclear whether it is a dish, so please input it," as shown in Fig. 6(c), for example. As a result, the correction unit 600 can accept an operation to input a correct dish, and set the reliability of the menu ID corresponding to the accepted dish to "1."

[0044] FIG. 7 is a diagram showing a feature map in an embodiment. The menu estimation system 1 may create a feature map in which the region of the target image and the label of the region are mapped for the target image. For example, assume that an egg dish estimation process is performed on a target image as shown in FIG. 7(a), and an egg dish is misidentified and not detected as shown in FIG. 7(b). In this case, the estimation unit 400 creates a feature map as shown in FIG. 7(d). This feature map shows an attention region corresponding to a dish that is misidentified as an egg dish and a label (L) of the attention region. In response to an operation to correct the region and label of an egg dish as shown in FIG. 7(c), for example, the correction unit 600 creates a corrected feature map # including the corrected attention region # and label # as shown in FIG. 7(e). As a result, the menu estimation system 1 outputs the corrected target image including the region information and the feature map # to the image storage unit 200, and the model learning unit 300 can perform a learning process using the corrected target image and the feature map #.

[0045] [Volume estimation process] Next, volume estimation in the embodiment will be described. Fig. 8 is a diagram showing an image of a food captured from above and the area. Fig. 9 is an image of a food captured from the side and the area. When the target image is an image of food captured from above as shown in Fig. 8, the size of each area recognized as a food menu item corresponds to the volume of the food, so estimation unit 400 can estimate the volume with high accuracy. In contrast, when ramen noodles with different volumes are captured from above as shown in Fig. 9, the volume estimation accuracy is low because there is a difference in volume in the height (depth) direction.

[0046] Furthermore, since the size of the food in the target image changes depending on the distance (depth) from the imaging device to the food, the volume estimation result changes depending on the distance (depth) from the imaging device to the food. Furthermore, since the size of the food in the target image changes depending on the viewpoint of the imaging device with respect to the food, the volume estimation result changes depending on the viewpoint of the imaging device with respect to the food. Furthermore, since the area recognized as food changes depending on the ratio of incident light to reflected light on the food and the lighting, the volume estimation result changes depending on the ratio of incident light to reflected light on the food and the lighting. From the above, the reliability of the label is determined by a function that takes the label and environmental information as input.

[0047] Therefore, as shown in Fig. 4, the menu estimation system 1 trains the volume estimation model 422 by including distance information D_i (i = 1, 2,..., N), environmental information E_i (i = 1, 2,..., N), and reliability Ri (i = 1, 2,..., N) in the training data in addition to image data X_i (i = 1, 2,..., N), region, and label. The environmental information E_i (i = 1, 2,..., N) is defined by viewpoint information: V_i (i = 1, 2,..., N), the ratio of incident light to reflected light: A_i (i = 1, 2,..., N), and lighting L_i (i = 1, 2,..., N). The reliability Ri is determined by a function F to which the lighting Li and environmental information Ei are input. 5, the menu estimation system 1 can output with high accuracy the area, distance, environment, and label reliability of the target image as the estimation result in response to inputting the target image into the volume estimation model 422. Specifically, as shown in FIG 9, the distance information D_i (i=1,2,3), environment information E_i (i=1,2,3), and reliability Ri (i=1,2,3) can be estimated for each area corresponding to the menu.

[0048] FIG. 10 is a block diagram showing an example of a nutrient estimation unit in an embodiment. The menu estimation system 1 may include a nutrient estimation unit 800. The nutrient estimation unit 800 is realized by a processor such as a CPU executing a program stored in a program memory. The nutrient estimation unit 800 estimates nutrients of a menu based on information on a food menu and the volume of the menu. Specifically, the nutrient estimation unit 800 inputs menu information 710 and volume information 720, and estimates nutrient information. The nutrient information is information including nutrients and reliability corresponding to the menu. The nutrient information may be information including nutrients and reliability for each area in the menu. The nutrient information may be the weight of the three major nutrients (protein, carbohydrate, and lipid) included in the menu.

[0049] [Data generation process] Next, the process of generating learning data by the learning data generating unit 210 will be described in detail. FIG. 11 is a diagram for explaining an example of the data generation process in the embodiment. FIG. 12 is a diagram for explaining another example of the data generation process in the embodiment. The learning data generation unit 210 generates a learning image (hereinafter also referred to as a generated image) by using a correct image as an original image. The correct image is, for example, an image that has received an operation from the user terminal device 100 indicating that the estimation result is correct, or an image to which a label with a reliability of a predetermined value or more is attached. As shown in FIG. 11, the learning data generation unit 210 generates a learning image by, for example, a process of superimposing an obstacle image on the correct image, a process of inverting the correct image, or a process of synthesizing a plurality of correct images. The obstacle image may be any image that makes it difficult to estimate anything other than the menu to be estimated. As shown in FIG. 12, the learning data generation unit 210 may generate a learning image by using a misrecognized image as an original image.

[0050] "First Algorithm" First, the first algorithm in the data generation process will be described. FIG. 13 is a diagram showing an example of an estimation result in an embodiment. An estimation result is generated in which the number of images is "1", the number of regions is "3", the number of labels is "3", and the labels are "egg dish, bread, and miso soup". Here, let the types of labels be L. For example, L = 1 is set to "ramen", L = 2 is set to "fried rice", and L = 3 is set to "egg dish". Also, let the number of original images be N, the number of generated images be M, and the total number of original images and generated images be N + M. Note that multiple labels may be associated with each target image or each region, and there may be a menu that is not labeled.

[0051] Let the i-th generated image in the set of generated images be O i (i = 1, 2, ···, N + M), and O i = {i} (i = 1, 2, ···, N) is represented, and O i = {a, b, c} (i = N + 1, N + 2, ···, N + M), 1 ≤ a < b < c ≤ N is represented. For example, when the (N + 1)-th generated image is generated from the 3rd, 5th, and 7th original images, the generated image is O N+1 = {3, 5, 7} is represented. Let the total value of the appearance numbers of the original images of each of the multiple generated images be T i (i = 1, 2, ···, N). For example, when there are 4 generated images of O1 = {1}, O2 = {2}, O3 = {3}, and O4 = {1, 3}, the original image of i = 1 is used 2 times, the original image of i = 2 is used 1 time, and the original image of i = 3 is used 2 times. Therefore, T1 = {2}, T2 = {1}, and T3 = {2}.

[0052] FIG. 14 is a diagram showing the operation in the learning data generation unit of the embodiment, and FIG. 15 is a diagram showing an example of the generation process of the learning image in the embodiment. The learning data generation unit 210 inputs a plurality of original images and generates learning images. The learning data generation unit 210 adjusts the number N of original images to be evenly close among the original images and generates learning images. For example, when the learning data generation unit 210 sets the average value of the used original images as (T hat) in the following formula 1, the learning images are generated so that the mean square error in the following formula 2 is below a certain value.

Equation

[0053] The learning data generating unit 210 may extract (sample) a set of learning images from all learning images such that the square error of the number of original images used is equal to or less than a certain value, and set the extracted set of learning images as learning data. Furthermore, the learning data generating unit 210 may extract a set of learning images that belong to a predetermined range from the median of the number of original images used in all learning images, and set the extracted set of learning images as learning data. The process of extracting a set of learning images is, for example, a process of reducing the number of learning images labeled with ramen when, after the generation of learning images, the ratio of learning images labeled with ramen, learning images labeled with fried rice, and learning images labeled with egg dishes is significantly biased toward one of the learning images, or when, for example, the number of learning images labeled with ramen is too large compared to the number of other learning images.

[0054] FIG. 16 is a diagram showing another example of the generation process of learning images in the embodiment. The learning data generation unit 210 may adjust the number of original images between the original images based on the menu of the estimation target (estimation target A). For example, when the estimation target is ramen and the menu estimation model 412 and / or the volume estimation model 422 are trained, the learning data generation unit 210 generates learning images mainly from original images with a label corresponding to ramen. Specifically, the learning data generation unit 210 generates a set of learning images using, as original images, a correct image with a label corresponding to ramen, a misidentified image with a label corresponding to ramen, an image with a high reliability of the label corresponding to ramen, an image with a low reliability of the label corresponding to ramen, and an image group with a label corresponding to ramen and various distances and environments.

[0055] The learning data generation unit 210 calculates a desirable ratio of learning images according to the estimation target by multiplying the number of original image occurrences T' i(i=1, 2, . . . , N). The ratio of the desired number of learning images may be set manually by the user or based on a predetermined criterion. For example, the number of occurrences of an original image to which the most desirable label (ramen) as an estimation target is added is set to T'1, and the number of occurrences of an original image to which the second most desirable label (fried rice) as an estimation target is added is set to T'2. The learning data generating unit 210 substitutes T'1 and T'2 into formula 3, and executes a process of generating learning images so that the square error obtained by formula 3 is equal to or smaller than a predetermined value, or a process of adjusting the number of learning images so that the square error obtained by formula 3 is equal to or smaller than a predetermined value. In addition, the learning data generating unit 210 may extract a set of learning images that belong to a predetermined range from the median value of the number of occurrences of original images in all learning images, and set the extracted set of learning images as learning data.

number

[0056] "Second Algorithm" 17 is a diagram showing another example of the generation process of learning images in the embodiment. i Let (i=1,2,...,N+M). For example, if the first learning image is labeled with "1 (ramen)" and "2 (fried rice)", then L1 = {1,2}. Label L i Expand the set of labels and count the total number of occurrences of each label as TL i (i = 1, 2, , N + M). For example, if there are three labels of “1” in the set of labels associated with 10 learning images, then TL1 = 3.

[0057] The learning data generation unit 210 adjusts the number of labels attached to the learning images between the learning images. For example, when the average value (TL hat) of the labels used in the original images is expressed by the following formula 4, the learning data generation unit 210 generates the learning images so that the square error as shown in the following formula 5 is equal to or less than a certain value.

number

[0058] The training data generation unit 210 may extract (sample) a set of training images from all training images such that the square error of the number of labels used is equal to or less than a certain value, and set the extracted set of training images as training data. Furthermore, the training data generation unit 210 may extract a set of training images that belong to a predetermined range from the median number of labels used in all training images, and set the extracted set of training images as training data. The process of extracting a set of training images is, for example, a process of reducing the number of training images to which the ramen label is added when, after training images are generated, the ratio of ramen labels, fried rice labels, and egg dish labels is significantly biased toward any of the generated images, or when, for example, the number of ramen labels is too large compared to the number of other labels.

[0059] FIG. 18 is a diagram showing another example of the generation process of learning images in the embodiment. The learning data generation unit 210 may adjust the number of labels attached to the learning images between the learning images based on the menu of the estimation target. For example, when the estimation target is ramen and the menu estimation model 412 and / or the volume estimation model 422 is trained, the learning data generation unit 210 generates learning images centered on the label corresponding to ramen. Specifically, the learning data generation unit 210 generates a set of learning images using, as source images, a correct image attached with a label corresponding to ramen, a misidentified image attached with a label corresponding to ramen, an image with a high reliability of the label corresponding to ramen, an image with a low reliability of the label corresponding to ramen, and a group of images attached with a label corresponding to ramen and with various distances and environments.

[0060] The learning data generation unit 210 calculates a desirable ratio of label appearances according to the estimation target as the label appearance number TL' i(i=1, 2, . . . , L). The ratio of the number of desirable label occurrences may be set manually by the user or based on a predetermined criterion. For example, the number of occurrences of the most desirable label (ramen) as an estimation target is set to TL'1, and the number of occurrences of the second most desirable label (fried rice) as an estimation target is set to TL'2. The learning data generation unit 210 substitutes TL'1 and TL'2 into formula 6, and executes a process of generating learning images so that the square error obtained by formula 6 is equal to or smaller than a predetermined value, or a process of adjusting the number of learning images so that the square error obtained by formula 6 is equal to or smaller than a predetermined value. The learning data generation unit 210 may also extract a set of learning images that belong to a predetermined range from the median value of the number of occurrences of the labels in all learning images, and set the extracted set of learning images as learning data.

number

[0061] "The Third Algorithm" The training data generation unit 210 may adjust the number of original images between the original images as in the first algorithm, and may adjust the number of labels attached to the training images between the training images as in the second algorithm. For example, the training data generation unit 210 generates training images such that the average value when the original images are used is expressed as Equation 1 and the square error as expressed as Equation 2 is equal to or less than a certain value, and generates training images such that the average value when the labels of the original images are used is expressed as Equation 4 and the square error as expressed as Equation 5 is equal to or less than a certain value.

[0062] The training data generation unit 210 may extract (sample) a set of training images from all training images such that the squared error of the number of original images used is equal to or less than a certain value, and set the extracted set of training images as training data, and may also extract (sample) a set of training images from all training images such that the squared error of the number of labels used is equal to or less than a certain value, and set the extracted set of training images as training data. Furthermore, the training data generation unit 210 may extract a set of training images that falls within a predetermined range from the median of the number of original images used in all training images, and set the extracted set of training images as training data, and may also extract a set of training images that falls within a predetermined range from the median of the number of labels used in all training images, and set the extracted set of training images as training data.

[0063] The learning data generating unit 210 may adjust the number of original images between the original images and adjust the number of labels attached to the learning images between the learning images based on the menu of the estimation target. For example, when the estimation target is ramen and the menu estimation model 412 and / or the volume estimation model 422 is to be trained, the learning data generating unit 210 generates learning images centered on original images corresponding to ramen and labels corresponding to ramen. Specifically, the learning data generating unit 210 generates a set of learning images using original images corresponding to ramen and correct images with labels attached, original images corresponding to ramen and misidentified images with labels attached, images with high reliability of the label corresponding to ramen, images with low reliability of the label corresponding to ramen, and images with labels corresponding to ramen and attached with various distances and environments as original images.

[0064] The learning data generation unit 210 calculates a desirable ratio of learning images and a desirable ratio of label occurrences according to the estimation target by dividing the original image occurrences T' i (i=1,2,...,N) and the label occurrence count TL' i(i=1, 2, . . . , L) may be used to set the number of occurrences of the original image of ramen, which is the first most desirable estimation target, to T'1, and the number of occurrences of the label (ramen) to TL'1, and the number of occurrences of the original image of fried rice, which is the second most desirable estimation target, to T'2, and the number of occurrences of the label (fried rice) to T'2. The learning data generation unit 210 substitutes T'1 and T'2 into formula 3 and TL'1 and TL'2 into formula 6, and executes a process of generating learning images so that each of the squared errors obtained by formula 3 is equal to or smaller than a predetermined value, or a process of adjusting the number of learning images so that each of the squared errors obtained by formula 6 is equal to or smaller than a predetermined value. The learning data generation unit 210 may also extract a set of learning images that belong to a predetermined range from the median value of the number of occurrences of the original image and the median value of the number of occurrences of the label in all learning images, and set the extracted set of learning images as learning data.

[0065] FIG. 19 is a diagram showing a specific example of adjusting a set of learning images. Specifically, it is assumed that the menu to be estimated is ramen, and a set A of learning images exists. The learning data generation unit 210 adjusts the number of occurrences of the label "ramen" to be higher than the number of occurrences of other labels using Equation 4 in order to increase the ratio of learning images with the label "ramen" in the set A of learning images, thereby extracting a set B of learning images from the set A of learning images. Next, the learning data generation unit 210 extracts a set C of learning images from the set B of learning images so as to adjust the learning images using the original image of ramen so that they are uniform. Alternatively, the learning data generation unit 210 extracts a set D of learning images from the set B of learning images so as to adjust the set to include learning images close to the original image of ramen.

[0066] [Selection of original image] Next, the process of selecting an original image will be described. This process of selecting an original image can be applied to any of the first algorithm, the second algorithm, and the third algorithm described above. FIG. 20 is a flowchart showing an example of a procedure of a process for generating a learning image. The learning data generating unit 210 acquires an original image from the image storage unit 200 (step S100). The learning data generating unit 210 determines whether or not the reliability associated with the acquired original image is equal to or greater than a threshold (step S102). If the reliability is equal to or greater than a threshold (step S102: YES), the learning data generating unit 210 generates a learning image using the original image (step S104), and if the reliability is not equal to or greater than the threshold (step S102: NO), the learning data generating unit 210 does not generate a learning image using the original image. In this way, the learning data generating unit 210 can select an original image by determining whether or not to generate a learning image using the original image based on the reliability corresponding to the label.

[0067] Specifically, we define the confidence level of the label associated with the i-th image as R i Assume that (i=1,2,...,N+M), the first original image is accompanied by labels "1 (ramen)" and "2 (fried rice)", the reliability of label "1" is 0.9, and the reliability of label "2" is 0.5, and L1={1,2}, R1={0.9,0.5}. In this case, for example, when the reliability threshold is set to 0.7 for all menus (all labels), if the first original image has L1={1,2}, R1={0.9,0.5}, and the second original image has L2={1,2}, R2={0.5,0.8}, the learning data generation unit 210 determines that the first original image is to be used as an original image for training the menu estimation model 412 and the volume estimation model 422 that predict ramen corresponding to the label "1" (0.9>0.7), and that the second original image is not to be used (0.5<0.7). On the other hand, the learning data generation unit 210 determines not to use the first original image as an original image for training the menu estimation model 412 and the volume estimation model 422 that predict fried rice corresponding to the label "2" (0.5<0.7), and determines to use the second original image (0.8>0.7). In this way, the learning data generation unit 210 determines whether or not to generate learning images using original images based on the reliability corresponding to the label, based on the menu to be estimated.

[0068] [Evaluation process of the estimated model] FIG. 21 is a block diagram showing the evaluation of an estimation model in an embodiment. The menu estimation system 1 includes an evaluation unit 900. The evaluation unit 900 inputs an evaluation image to the menu estimation model 412 or the volume estimation model 422, and evaluates the menu estimation model 412 or the volume estimation model 422 based on the output of the menu estimation model 412 or the volume estimation model 422. The prediction accuracy of the menu estimation model 412 and the volume estimation model 422 changes depending on the reliability of the label attached to the original image. For example, the prediction accuracy of an estimation model trained using a set of original images in which the reliability of the label "1" is high but the reliability of the label "2" is low is often high for the label "1" but low for the label "2". When the menu estimation system 1 wants to increase the evaluation value for the label of the estimation model, it can generate an estimation model by selecting a label with high reliability in the above-mentioned selection of the original image. On the other hand, when the prediction accuracy of the label of the estimation model is acceptable even if it is low, the menu estimation system 1 can generate an estimation model by setting a low threshold for the reliability in the above-mentioned selection of the original image.

[0069] [Effects of the embodiment] As described above, according to the menu estimation system 1 of the embodiment, a menu estimation device can be realized that includes the image storage unit 200 as an image acquisition unit that acquires an image, the model learning unit 300 that inputs the image acquired by the image storage unit 200 as learning data and trains the menu estimation model 412 to output the menu of the food included in the image acquired by the image storage unit 200, the menu estimation unit 410 that inputs the image acquired by the image storage unit 200 as a target image to the menu estimation model 412 and estimates the menu of the food included in the target image based on the output of the menu estimation model 412, and the intermediate output unit 500 that outputs first menu information indicating the menu estimated by the menu estimation unit 410 and first region information indicating the image region in the target image corresponding to the first menu information. According to this menu estimation system 1, since the first menu information indicating the estimated menu and the first region information indicating the image region in the target image corresponding to the first menu information are output, it is possible to correct the accuracy of the region corresponding to the menu, and the estimation accuracy of the menu estimation model 412 can be improved based on the correction result.

[0070] Moreover, menu estimation system 1 includes correction unit 600 that receives an instruction to modify the first menu information and first region information output from intermediate output unit 500 and corrects the menu and image region based on the received instruction, and output unit 700 that outputs second menu information indicating the menu corrected by correction unit 600 and second region information indicating the image region, and allows model learning unit 300 to learn menu estimation model 412 based on the first menu information, first region information, second menu information, and second region information. As a result, menu estimation system 1 can improve the accuracy of menu estimation.

[0071] Furthermore, according to the menu estimation system 1, the estimation unit estimates a menu included in the target image and a part included in the menu, and outputs character information indicating the menu and character information indicating the part included in the menu. Furthermore, according to the menu estimation system 1, when the estimation unit estimates multiple menus for one image area included in the target image, information indicating the image area, character information indicating each menu and the reliability of each menu, and character information indicating a prompt to select one of the multiple menus are output. Furthermore, according to the menu estimation system 1, when the estimation unit cannot estimate a menu for an image area included in the target image, information indicating the image area and character information indicating a prompt to input a menu are output. As a result, according to the menu estimation system 1, in addition to the estimated menu, the image area on which the estimation was based can be presented to the user, the estimation result can be appropriately corrected, and the accuracy of the menu estimation can be further improved.

[0072] Furthermore, according to menu estimation system 1, an image is input as learning data, volume estimation model 422 is trained to output information about the menu of food items included in the image and the volume of the menu, and an image is input as a target image to volume estimation model 422, and information about the menu of food items and the volume of the menu is estimated based on the output of volume estimation model 422. This allows menu estimation system 1 to estimate the volume included in the menu.

[0073] Furthermore, according to menu estimation system 1, the information relating to volume includes information on the distance from the imaging device that captured the target image to the food, the viewpoint of the imaging device relative to the food, the incident light and reflected light relative to the food, and information on the lighting, so that the accuracy of volume estimation can be improved by taking into account the distance, viewpoint, lighting environment, and lighting that affect the size of the area included in the target image.

[0074] Furthermore, the menu estimation system 1 includes a nutrient estimation unit that estimates the nutrients of a menu based on information about a food menu and the volume of the menu, and can provide information about nutrients for each region based on the estimated menu and volume.

[0075] According to the menu estimation system 1 of the embodiment, a learning device can be realized that includes an image storage unit 200 as an image acquisition unit that acquires images, a learning data generation unit 210 as an image generation unit that generates learning images using the images acquired by the image storage unit 200 as original images, the learning data generation unit 210 adjusting the number of original images between the original images, and a learning processing unit 310 that inputs the images acquired by the image storage unit 200 and the learning images generated by the learning data generation unit 210 as learning data and trains an estimation model (menu estimation model 412 or volume estimation model 422) to output a menu of foods included in the learning data. According to this menu estimation system 1, it is possible to prevent the learning of the estimation model from being biased toward a specific original image.

[0076] Furthermore, according to the menu estimation system 1, the number of original images is adjusted between the original images based on the menu to be estimated, so that the estimation model can be trained according to the menu to be estimated, thereby improving the estimation accuracy of the estimation model.

[0077] Furthermore, menu estimation system 1 includes image storage unit 200 as an image acquisition unit that acquires images, learning data generation unit 210 as an image generation unit that generates learning images using the images acquired by image storage unit 200 as original images, and a learning unit that inputs the images acquired by image storage unit 200 and the learning images generated by image storage unit 200 as learning data and trains an estimation model to output menus of foods included in the learning data, and learning data generation unit 210 adjusts the number of labels associated with the learning images between the learning images. As a result, menu estimation system 1 can prevent learning of the estimation model from being biased toward a specific label.

[0078] Furthermore, according to the menu estimation system 1, the learning data generation unit 210 adjusts the number of labels attached to the learning images between the learning images based on the menu to be estimated, so that the estimation model can be trained according to the menu to be estimated, and the estimation accuracy of the estimation model can be improved.

[0079] Furthermore, according to the menu estimation system 1, the number of original images is adjusted between the original images, and the number of labels attached to the learning images is adjusted between the learning images, so that it is possible to prevent the learning of the estimation model from being biased toward a specific original image or a specific label.

[0080] Furthermore, according to the menu estimation system 1, the number of original images is adjusted between the original images based on the menu to be estimated, and the number of labels attached to the learning images is adjusted between the learning images, so that the estimation model can be trained according to the menu to be estimated, and the estimation accuracy of the estimation model can be improved.

[0081] Furthermore, according to the menu estimation system 1, whether or not to generate a learning image using the original image is determined based on the reliability corresponding to the label, so that the estimation model can be trained using an original image with a highly reliable label, thereby further improving the accuracy of menu estimation.

[0082] Furthermore, according to the menu estimation system 1, a determination is made based on the menu to be estimated and the reliability corresponding to the label as to whether or not to generate a learning image using the original image. The estimation model can be trained using an original image with a highly reliable label attached to the menu to be estimated, thereby further improving the accuracy of menu estimation.

[0083] Furthermore, the menu estimation system 1 includes an evaluation unit 900 that inputs an evaluation image accompanied by a label and reliability corresponding to the menu into an estimation model and evaluates the estimation model based on the output result of the estimation model, and can evaluate the estimation model based on a comparison between the output of the estimation model and the label accompanied by the evaluation image and the reliability accompanied by the evaluation image, thereby making it possible to create a highly rated estimation model.

[0084] Although each embodiment and each variant have been described, these are merely examples and are not intended to be limiting. For example, any of the embodiments or variants, or a part of each embodiment or a part of each variant, may be combined with one or more other embodiments or one or more other variants to realize one aspect of the present invention.

[0085] In addition, programs for executing each process of the user terminal device 100, the learning data generation unit 210, the model learning unit 300, the estimation unit 400, and the correction unit 600 in this embodiment may be recorded on a computer-readable recording medium, and the programs recorded on the recording medium may be read into and executed by a computer system, thereby performing the various processes described above related to the user terminal device 100, the learning data generation unit 210, the model learning unit 300, the estimation unit 400, and the correction unit 600.

[0086] Note that the "computer system" referred to here may include hardware such as the OS and peripheral devices. Furthermore, if a WWW system is used, the "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, writable non-volatile memories such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

[0087] Furthermore, the term "computer-readable recording medium" refers to the volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that is a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. This includes devices that hold a program for a certain period of time, such as a random access memory (Random Access Memory). The program may be transmitted from a computer system in which the program is stored in a storage device to another computer system via a transmission medium, or by a transmission wave in the transmission medium.

[0088] Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system, a so-called difference file (difference program). [Explanation of symbols]

[0089] 1. Menu Estimation System 100 User terminal device 200 Image storage unit 206 Log data storage unit 208 Detection result storage unit 210 Learning Data Generation Unit 300 Model Learning Department 310 Learning processing unit 400 Estimation section 410 Menu Estimation Department 412 Menu Estimation Model 420 Volume Estimation Unit 422 Volume Estimation Model 500 Intermediate output section (first output section) 510 Menu Information 520 Volume Information 600 Correction Unit 610 Menu correction section 620 Volume Correction Unit 700 Output section (second output section) 710 Menu Information 720 Volume Information 800 Nutrient Estimation Section 900 Evaluation Department

Claims

1. an image acquisition unit that acquires an image; a learning unit which uses an image acquired by the image acquisition unit as learning data, trains a menu estimation model to output information about a menu of food included in the image acquired by the image acquisition unit, and trains a volume estimation model to output information about the volume of the menu; the information about the volume includes information about a distance from an imaging device that captured the image acquired by the image acquisition unit to the food, a viewpoint of the imaging device with respect to the food, light incident on and reflected from the food, and environmental information about lighting; an estimation unit that inputs an image acquired by the image acquisition unit as a target image into the menu estimation model, estimates information about a menu of food included in the target image based on an output of the menu estimation model, and outputs first menu information and a reliability level related to the menu estimated by the menu estimation model, inputs an image acquired by the image acquisition unit as a target image into the volume estimation model, estimates information about the volume of the menu based on the output of the volume estimation model, and outputs first volume information about the volume of the menu estimated by the volume estimation model and a reliability level determined by a function to which information about a viewpoint of the imaging device, incident light and reflected light on food, and lighting are input; a first output unit that outputs the first menu information and the first volume information estimated by the estimation unit, and first region information indicating an image region in the target image corresponding to the first menu information and the first volume information; a correction unit that receives an instruction to correct the first menu information, the first volume information, and the first region information output from the first output unit, and corrects the menu and the reliability corresponding to the menu, the image region, the distance information, and the environment information based on the received instruction; a second output unit that outputs second menu information indicating information related to the menu corrected by the correction unit, a reliability corresponding to the menu, second region information indicating the image region, the distance information, and the environment information; a learning data generating unit that determines whether or not to generate a learning image using the original image based on the reliability corresponding to the menu and the reliability corresponding to the volume associated with the original image, and generates an image as learning data; the learning unit trains the menu estimation model using the image generated by the training data generation unit, the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu, and trains the volume estimation model using the first volume information, the image as the generated training data, the first region information, the second region information, the corrected distance information, the corrected environmental information, and the reliability corresponding to the volume; Menu estimator.

2. The estimation unit estimates a menu included in the target image and a portion included in the menu, The menu estimation device according to claim 1 , wherein the first output unit outputs character information indicating the menu and character information indicating a portion included in the menu.

3. 2. The menu estimation device according to claim 1, wherein, when a plurality of menus are estimated for one image region included in the target image, the estimation unit outputs information indicating the image region, text information indicating each menu and a reliability of each menu, and text information indicating a prompt to select one of the plurality of menus.

4. The menu estimation device according to claim 1 , wherein, when a menu cannot be estimated for an image area included in the target image, the estimation unit outputs information indicating the image area and text information prompting for input of a menu.

5. a nutrient estimation unit that estimates nutrients of the menu based on information about the food menu and the volume of the menu; The menu estimation device of claim 1 .

6. acquiring an image; a step of inputting the acquired image as learning data, training a menu estimation model so as to output information about the menu of food included in the acquired image, and training a volume estimation model so as to output information about the volume of the menu, the information about the volume including information about the distance from an imaging device that captured the acquired image to the food, a viewpoint of the imaging device with respect to the food, light incident on and reflected from the food, and environmental information about lighting; inputting the acquired image as a target image into the menu estimation model, estimating information regarding a menu of food included in the target image based on an output of the menu estimation model, and outputting first menu information and a reliability regarding the menu estimated by the menu estimation model; inputting the acquired image as a target image into the volume estimation model, estimating information on the volume of the menu based on an output of the volume estimation model, and outputting a reliability determined by a function to which first volume information on the volume of the menu estimated by the volume estimation model and information on the viewpoint of the imaging device, the incident light and reflected light on the food, and the lighting are input; outputting the estimated first menu information and the estimated first volume information, and first region information indicating an image region in the target image corresponding to the estimated first menu information and the estimated first volume information; receiving an instruction to modify the output first menu information, the first volume information, and the first region information, and correcting the menu and the reliability corresponding to the menu, the image region, the distance information, and the environment information based on the received instruction; outputting second menu information indicating information related to the corrected menu and a reliability corresponding to the menu, second region information indicating the image region, the distance information, and the environment information; determining whether or not to generate a learning image using the original image based on the reliability corresponding to the menu and the reliability corresponding to the volume associated with the original image, and generating an image as learning data; Including, the step of training an estimation model includes training the menu estimation model using an image as the generated training data, the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu, and training the volume estimation model using the first volume information, an image as the generated training data, the first region information, the second region information, the corrected distance information, the corrected environmental information, and the reliability corresponding to the volume; Menu estimation method.

7. On the computer, acquiring an image; a step of training an estimation model to output information regarding the menu of food included in the acquired image and the volume of the menu, using the acquired image as training data and training the menu estimation model to output information regarding the menu of food included in the acquired image and the volume of the menu, the information regarding the volume including information regarding the distance from an imaging device that captured the acquired image to the food, a viewpoint of the imaging device with respect to the food, incident light and reflected light on the food, and environmental information regarding lighting, inputting the acquired image as a target image into the menu estimation model, estimating information regarding a menu of food included in the target image based on an output of the menu estimation model, and outputting first menu information and a reliability regarding the menu estimated by the menu estimation model; inputting the acquired image as a target image into the volume estimation model, estimating information on the volume of the menu based on an output of the volume estimation model, and outputting a reliability determined by a function to which first volume information on the volume of the menu estimated by the volume estimation model and information on the viewpoint of the imaging device, the incident light and reflected light on the food, and the lighting are input; outputting the estimated first menu information and the estimated first volume information, and first region information indicating an image region in the target image corresponding to the estimated first menu information and the estimated first volume information; receiving an instruction to modify the output first menu information, the first volume information, and the first region information, and correcting the menu and the reliability corresponding to the menu, the image region, the distance information, and the environment information based on the received instruction; outputting second menu information indicating information related to the corrected menu and a reliability corresponding to the menu, second region information indicating the image region, the distance information, and the environment information; determining whether or not to generate a learning image using the original image based on the reliability corresponding to the menu and the reliability corresponding to the volume associated with the original image, and generating an image as learning data; Including, The step of training the estimation model includes training the menu estimation model using an image as generated training data, the first menu information, the first region information, the second menu information, the second region information, and the reliability corresponding to the menu, and training the volume estimation model using the first volume information, an image as generated training data, the first region information, the second region information, the corrected distance information, the corrected environmental information, and the reliability corresponding to the volume, the program.

Citation Information

Patent Citations

  • Object recognition system, object recognition method, and object recognition robot

    JP2008123365A

  • Information processing device, information processing system, and storage medium storing program

    JP2015005172A

  • Image processing device, image processing method, and image processing program

    JP2017211691A

  • Information processing unit, information processing method, and program

    JP2018185795A