Program, computer, system, and information processing method

Abstract

To provide a novel program, a server (computer) 10, a system, and an information processing method which grant information data regarding cooking to image data regarding cooking.SOLUTION: There are provided image accepting means 14 and data adding means 36. The image accepting means 14 accepts, from a photographing device 60, image data regarding cooking photographed by the photographing device 60. The data adding means 36 adds, to the image data regarding cooking, information data of at least either of cooking position information, cooking kind information, and cooking mass information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a program, a computer, a system, and an information processing method. [Background technology]

[0002] Conventionally, there have been various systems that acquire images of food using a mobile device such as a smartphone and automatically calculate the nutritional components of the food, such as calories (see, for example, Patent Document 1, etc.). In such systems, a learning model is generated by machine learning using, for example, images of food and types of food as training data, and this learning model is used to determine the type of food from the image of food acquired on the mobile device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Patent Application Publication WO2018 / 008686A1 Summary of the Invention [Problem to be solved by the invention]

[0004] To generate the learning model using machine learning, a dataset of food images annotated with various information about the food (e.g., type of food, etc.) is required. However, if such annotating of various information about the food is done manually, there is a problem that the workload becomes heavy.

[0005] The present disclosure has been made in consideration of these points, and aims to provide a new program, computer, system, and information processing method that adds data about food to data about images of food. [Means for solving the problem]

[0006] The program of the present disclosure is A program that causes a computer to function as an image receiving means and a data providing means, the image receiving means receives, from the imaging device, data of an image relating to food captured by the imaging device; The data providing means provides at least one of information on the position of the food, information on the type of food, and information on the mass of the food to the data on the image of the food.

[0007] In the program of the present disclosure, The data providing means may provide data of preset menu information to data of images relating to dishes.

[0008] The program of the present disclosure is The computer further functions as a plate position acquisition means and a food position acquisition means, the plate position acquisition means acquires information about the position of the plate on a support member on which the plate carrying food is placed, The food position acquisition means may acquire information about the position of the food from the acquired information about the position of the plate based on preset information about the food to be placed on the plate.

[0009] In the program of the present disclosure, The plate position acquisition means may receive, from the plate-receiving member, information about the position of the plate that the plate-receiving member has acquired from a wireless tag provided on the plate.

[0010] In the program of the present disclosure, The shape of the plate's contact surface is preset, The plate position acquisition means may receive information regarding the shape of the contact surface of the plate placed on the support member from the support member, and acquire information regarding the position of the plate on the support member from the received information regarding the shape of the contact surface of the plate.

[0011] The program of the present disclosure is causing the computer to further function as pressure distribution information acquisition means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; The plate position acquisition means may identify the shape of the contact surface of the plate placed on the plate-receiving member from the pressure distribution information acquired by the pressure distribution information acquisition means.

[0012] The program of the present disclosure is causing the computer to further function as contact distribution information acquisition means, the contact distribution information acquiring means acquires, from the support member, information on the contact distribution of members in contact with the support member; The plate position acquisition means may identify the shape of the contact surface of the plate placed on the support member from the contact distribution information acquired from the contact distribution information acquisition means.

[0013] The program of the present disclosure is causing the computer to further function as a light-shading distribution information acquisition unit; the light-blocking distribution information acquisition means acquires, from the support member, information on the light-blocking distribution of a member that is blocking the support member; The plate position acquisition means may identify the shape of the contact surface of the plate placed on the support member from the information on the light-blocking distribution acquired from the light-blocking distribution information acquisition means.

[0014] The program of the present disclosure is The computer further functions as a food type identification means, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, The food type identification means may receive information regarding the shape of the contact surface of a plate placed on the supporting member from the supporting member, identify the type of plate from the shape of the contact surface of the plate, and identify the type of food to be placed on the plate from the information on the identified type of plate.

[0015] The program of the present disclosure is The computer is further caused to function as a pressure distribution information acquisition means and a plate position acquisition means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; The plate position acquisition means may identify the shape of the contact surface of the plate placed on the plate-receiving member from the pressure distribution information acquired by the pressure distribution information acquisition means.

[0016] The program of the present disclosure is The computer functions as a food mass determination means, The food mass specifying means may specify the mass of the food placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means.

[0017] The program of the present disclosure is The computer is further caused to function as a dish shape specifying means and a dish volume predicting means, the dish shape identification means identifies the type of dish from the image of the food captured by the imaging device, and identifies the shape of the dish corresponding to the identified type of dish; the food volume prediction means predicts the volume of food placed on the plate from the identified shape of the plate; The data providing means may also provide data on predicted dish volume information to the data on the image of the dish.

[0018] The program of the present disclosure is The computer is further configured to function as a pressure distribution information acquisition means and a cooking content prediction means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; The cooking content prediction means may predict at least one of the position of the food, the type of food, and the mass of the food from the pressure distribution information acquired from the pressure distribution information acquisition means and the image of the food captured by the imaging device.

[0019] The program of the present disclosure is causing the computer to further function as a pressure distribution information acquiring means, a converting means, and an image synthesizing means; the pressure distribution information acquiring means acquires information on the pressure distribution applied to a support member on which a plate with food placed thereon is placed, the converting means converts the two-dimensional image generated from the pressure distribution information acquired by the pressure distribution information acquiring means so that the image matches the shape of the support member in the image accepted by the image accepting means; The image synthesis means may generate a synthetic image by superimposing the converted pressure distribution image on the image accepted by the image acceptance means.

[0020] The computer of the present disclosure includes: A computer that functions as an image receiving means and a data providing means by executing a program, the image receiving means receives, from the imaging device, data of an image relating to food captured by the imaging device; The data providing means provides at least one of information on the position of the food, information on the type of food, and information on the mass of the food to the data on the image of the food.

[0021] The system of the present disclosure comprises: A computer, a support member on which a plate carrying food is placed; A system comprising: the support member has a detection unit that detects information on a pressure distribution applied to the support member, The computer executes a program to function as an image receiving means, a pressure distribution information acquiring means, a plate position acquiring means, a food position acquiring means, and a data assigning means, the image receiving means receives, from the imaging device, image data of the placement member and the food placed on a plate placed on the placement member, the image data being captured by the imaging device; the pressure distribution information acquisition means acquires, from the support member, information on the pressure distribution applied to the support member detected by the detection unit; the dish position acquisition means identifies the shape of the contact surface of the dish placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means, and acquires information about the position of the dish on the support member from the information about the identified shape of the contact surface of the dish; the food position acquisition means acquires food position information from the acquired dish position information based on preset information about the food to be placed on the plate; The data providing means provides at least data on the location of the food to the data on the image of the food.

[0022] In the system of the present disclosure, The computer further functions as a cuisine type identification means by executing the program, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, the food type identification means identifies the type of dish from the shape of the contact surface of the identified dish, and identifies the type of food to be placed on the dish from information on the identified type of dish; The data providing means may also provide data on the type of food to the data on the image relating to the food.

[0023] In the system of the present disclosure, The computer further functions as a food mass determination means by executing the program, the food mass specifying means specifies the mass of the food to be placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means; The data providing means may also provide data on the mass of the food to the data on the image of the food.

[0024] The system of the present disclosure also includes: A computer, a support member on which a plate carrying food is placed; A system comprising: the support member has a detection unit that detects information on a pressure distribution applied to the support member, The computer executes a program to function as a pressure distribution information acquisition means and a plate position acquisition means, the pressure distribution information acquisition means acquires, from the support member, information on the pressure distribution applied to the support member detected by the detection unit; The plate position acquiring means acquires information about the position of the plate on the receiving member from the pressure distribution information acquired by the pressure distribution information acquiring means.

[0025] In the system of the present disclosure, The plate position acquisition means may identify the shape of the contact surface of the plate placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means, and acquire information regarding the position of the plate on the support member from the information regarding the identified shape of the contact surface of the plate.

[0026] In the system of the present disclosure, The computer further functions as a cuisine type identification means by executing the program, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, The food type identification means may identify the type of dish from the shape of the contact surface of the identified dish, and identify the type of food to be placed on the dish from information on the identified type of dish.

[0027] In the system of the present disclosure, The computer further functions as a food mass determination means by executing the program, The food mass specifying means may specify the mass of the food to be placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means.

[0028] The information processing method of the present disclosure includes: An information processing method performed by a computer having a control unit, the control unit receives, from an imaging device, data of an image of food captured by the imaging device; The control unit is characterized in that it adds at least one of information data of the position of the food, information of the type of food, and information of the mass of the food to the data of the image relating to the food. [Effects of the Invention]

[0029] According to the present disclosure, it is possible to provide a new program, computer, system, and information processing method for adding data about information related to a dish to data about an image of a dish. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a diagram schematically illustrating an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a diagram illustrating an example of information stored in a storage unit of a computer according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram showing an example of a preset correspondence relationship between dish shape, specification (type), weight, capacity, and installation surface shape. [Figure 4] FIG. 10 is a diagram showing an example in which information data relating to the type, position, and mass of food is annotated to image data. [Figure 5] FIG. 10 is a diagram showing an example in which data on the type of food, the type of plate, and the pressure distribution of the plate are annotated to image data. [Figure 6] FIG. 10 is a diagram showing an example in which menu information data is annotated to image data. [Figure 7] FIG. 10 illustrates an example of a projective transformation according to an embodiment of the present disclosure. [Figure 8] 1A and 1B are diagrams illustrating features of a receiving member according to an embodiment of the present disclosure compared to a prior art; [Figure 9] FIG. 1 is a diagram illustrating an exemplary information processing flow of an information processing method according to an embodiment of the present disclosure. [Figure 10]FIG. 10 is a diagram showing another exemplary flow of information processing of the information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] 1 to 10 are diagrams illustrating an information processing system 1 and an information processing method according to the present disclosure.

[0032] [Information Processing System 1] An information processing system 1 according to this embodiment includes a server 10 and a receiving member 50. As shown in Fig. 1, the information processing system 1 may include an imaging device 60, and may further include a management terminal 46 and a staff terminal 48 for operating the server 10. The server 10 is a management server (computer) that can send and receive information to and from the receiving member 50, imaging device 60, management terminal 46, etc. via a communication network (not shown) such as the Internet line.

[0033] <Server (Computer) 10> The server 10 can be configured as an industrial computer, a personal computer, a tablet terminal, or the like, or can be a virtual server within a physical server accessible via the Internet. There may be multiple pieces of hardware or virtual servers, or any combination of these may be used. The server 10 shown in FIG. 1 includes a control unit 12, a storage unit 42, and a communication unit 44.

[0034] (Control unit 12) The control unit 12 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and controls the operation of the server 10. Specifically, the control unit 12 executes programs stored in a storage unit 42 (described later) to function as an image receiving unit 14, a pressure distribution information acquiring unit 16, a contact distribution information acquiring unit 18, a shading distribution information acquiring unit 20, a plate position acquiring unit 22, a food position acquiring unit 24, a food type identifying unit 26, a food mass identifying unit 28, a plate shape identifying unit 30, a food volume predicting unit 32, a food content predicting unit 34, a data assigning unit 36, a conversion unit 38, an image combining unit 40, and the like. Note that these functions may be achieved by executing one or more independent programs or applications. Furthermore, these programs and applications may be provided on a single terminal (the server 10) or distributed across multiple terminals (including the server 10), and in the latter case, may be connected to each other via a wired cable or a communication network. Each unit will be described later.

[0035] (Storage unit 42) The storage unit 42 is configured, for example, with an HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), etc. Furthermore, the storage unit 42 is not limited to being built into the server 10, but may be a storage medium (for example, a USB memory) that can be detachably attached to the server 10. Furthermore, instead of providing the storage unit 42, various pieces of information may be stored in other storage means (such as a cloud server).

[0036] Examples of information and data stored in the memory unit 42 include data on the captured image of food (which may include the support member 50 and the plate) received from the imaging device 60 and processed data such as conversion thereof, information received from the support member 50 (pressure distribution, contact distribution, shading distribution, and the shape of the support surface of the placed plate and the position of the plate derived from these), and various data related to the captured food (position, type, mass, volume, etc. of the food).In addition, the memory unit 42 can also store information on the food to be placed on a plate that is set in advance prior to capturing the image of the food, information on the shape of the plate that is set in advance (including 3D information, volume, and the shape of the support surface of the plate) (plate specifications and type), the relationship between the preset type of plate and the shape of the support surface of the plate, and the relationship between the preset type of plate and the type of food to be placed on the plate, etc. Generally, the memory unit 42 can store the following information: the support member information (support member identification information, structural information, detector information, etc.) for the support member 50 as shown in FIGS. 1 and 2; the plate information (plate type identification information, plate mass information, contact surface shape, etc.) for the plate on which the imaged food is placed; the menu information (daily menu information including breakfast, lunch, dinner, etc., mass information for each dish); and the serving information (the type and number of dishes placed on the support member 50 for each date, the food information for each dish for each date, etc.). Also, FIG. 3 shows an example of a correspondence relationship between the shape, specifications (type), weight, capacity, and installation surface shape of the plate that is preset. Note that, in this application, the term "food" has the usual meaning and includes, but is not limited to, staple foods such as rice and bread, noodles, rice bowls, soups, unprocessed foods such as fruits and raw vegetables, beverages, and confectioneries. The term "menu" is a broad concept that includes at least one such dish.

[0037] (Communications Department 44) The communication unit 44 connects the control unit 12 to an external device (for example, the mounting member 50) wirelessly or via a wire so that the control unit 12 can communicate with the external device. The control unit 12 can send and receive information to and from the external device via the communication unit 44.

[0038] (others) In addition, the server 10 may be provided with an operation unit (input unit) such as a keyboard that enables an administrator of the server 10 to give various commands to the control unit 12, and a display unit (output unit) such as a monitor that displays various screens in response to a display command signal from the control unit 12. In one embodiment, a display operation unit such as a touch panel that integrates the operation unit and display unit may be used.

[0039] (Details of control unit 12) (Image reception means 14) The image receiving means 14 receives data of images of food captured by an imaging device 60 (described later) from the imaging device 60. The image receiving means 14 may also receive data of images of the placement member 50 and the food placed on a plate placed on the placement member 50 captured by the imaging device 60 (i.e., images of the placement member 50, the plate thereon, and the food).

[0040] (Pressure distribution information acquisition means 16, contact distribution information acquisition means 18, shading distribution information acquisition means 20) The pressure distribution information acquiring means 16 acquires information on the pressure distribution applied to the support member 50 from the support member 50. That is, the pressure distribution information acquiring means 16 acquires, from the support member 50 on which a plate with food on it is placed, information on the pressure distribution applied to the support member 50 by the plate with food on it. In the example shown in FIG. 1 , the pressure distribution information acquiring means 16 acquires, from the support member 50, information on the pressure distribution applied to the support member 50 detected by the detector 52 of the support member 50. The contact distribution information acquiring means 18 acquires, from the support member 50, information on the contact distribution of members (e.g., plates, chopsticks, packaged beverages, etc.) in contact with the support member 50. The shading distribution information acquiring means 20 acquires, from the support member 50, information on the shading distribution of members shading the support member 50. The position and contact surface shape of each plate can be identified from the pressure distribution, contact distribution, and shading distribution. Note that care should be taken to exclude information from items other than plates, such as the diner's hands, chopsticks, or spoon, which may be placed on the support member 50 and affect the pressure distribution, etc. The manner in which the pressure distribution, contact distribution, and shading distribution are acquired by the detection unit 52 of the support member 50 will be described later.

[0041] (Dish position acquisition means 22) The plate position acquisition means 22 acquires information relating to the position of the plate on the support member 50 on which the plate with food placed thereon is placed, from the support member 50. As will be described later, this plate position information can be considered as position information of the food. The plate position acquisition means 22 may also acquire plate position information from a source other than the support member 50. For example, the plate position acquisition means 22 may analyze an image captured by the imaging device 60 to directly acquire plate position information. This image analysis method is not particularly limited, and various image recognition and image analysis techniques can be used, and manual confirmation may also be performed.

[0042] The plate position acquisition means 22 can also receive information about the shape of the contact surface of a plate placed on the support member 50 from the support member 50. Here, if the shape of the plate's contact surface (i.e., the plate's specifications; see FIG. 3) is preset, the plate can be individually identified. For example, as shown in FIG. 3, presetting the correspondence between the plate's specifications (type) and the plate's contact surface shape is particularly suitable when the plates used are limited or when dietary and nutritional information is highly important (hospitals, rehabilitation facilities, elderly care facilities, etc.). Furthermore, the plate position acquisition means 22 acquires information about the position of the plate on the support member 50 (specific coordinates such as the edge or center of the support member 50) from the received information (coordinate range) about the "shape of the contact surface of a plate placed on the support member 50." In this way, the plate position acquisition means 22 can identify the type and position of the plate.

[0043] The plate position acquisition means 22 can identify the shape of the contact surface of the plate placed on the plate-receiving member 50 from the pressure distribution information acquired from the pressure distribution information acquisition means 16 (or the contact distribution information acquired from the contact distribution information acquisition means 18, or the shading distribution information acquired from the shading distribution information acquisition means 20). The plate position acquisition means 22 can also acquire information about the position of the plate on the plate-receiving member 50 from the pressure distribution (contact distribution or shading distribution) information.

[0044] The plate position acquisition means 22 can also receive, from the plate-receiving member 50, information about the position of the plate that the plate-receiving member 50 has acquired from a wireless tag attached to the plate. The wireless tag will be described later together with the plate-receiving member 50. In addition, if the relationship between the wireless tag attached to the plate and the shape of the plate's contact surface is set in advance, when the plate-receiving member 50 acquires information about the shape of the plate's contact surface via the wireless tag, the plate position acquisition means 22 may acquire information about the position of the plate based on the information about the shape of the plate's contact surface.

[0045] (Cooking position acquisition means 24) The food position acquisition means 24 acquires the food position information from the acquired dish position information based on the preset information of the food placed on the plate. Note that the food position acquisition means 24 may acquire the food position information directly from the image captured by the imaging device 60, for example, without relying on the dish position acquisition means 22.

[0046] (Means for identifying food type 26) When the relationship between the type of plate and the shape of the plate's contact surface is set in advance, and the relationship between the type of plate and the type of food to be placed on the plate is also set in advance, the food type identification means 26 identifies the type of plate from the shape of the contact surface of the plate placed on the placement member 50, and identifies the type of food to be placed on the plate from information on the identified food type. The food type identification means 26 may receive information on the shape of the plate's contact surface from the placement member 50.

[0047] (Means for identifying the amount of food 28) The food mass determining means 28 determines the mass of the food placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means 16. The food mass determining means 28 may receive information about the mass added to the plate-receiving member 50 at the location where the plate is located from the plate-receiving member 50 on which the plate with food placed on it is placed, and determine the mass of the food placed on the plate by subtracting a preset mass of the plate from the received mass information.

[0048] (Dish shape identification means 30) The dish shape identification means 30 identifies the type of dish from an image of food captured by the imaging device 60, and identifies the shape of the dish corresponding to the identified type of dish. At this time, if the type of dish and the shape of the dish (3D structure such as the depth of the dish, and size) are associated in advance, identification can be performed efficiently. Note that the method for identifying the type of dish from an image is not particularly limited, and various image recognition and image analysis techniques can be used, and manual confirmation work may also be performed. As described above, the type of dish on the placement member 50 may be set in advance.

[0049] (Means for predicting food volume 32) The food volume prediction means 32 predicts the volume of food placed on a plate from the identified plate shape. If the type of plate and its 3D structure data are associated in advance, the type of plate, i.e., the volume of food placed on the plate, can be predicted with high accuracy from the 3D structure data.

[0050] (Method 34 for predicting food content) The dish content prediction means 34 predicts at least one of the position, type, and mass of the dish from the pressure distribution information acquired from the pressure distribution information acquisition means 16 and the image of the dish captured by the imaging device 60. The dish content prediction means 34 can also predict the position, type, mass, volume, etc. of the dish from the image of the dish captured by the imaging device 60.

[0051] The above predictions can be made using a trained model generated by machine learning, without using feature quantities such as the "position of the plate" or the "shape of the plate's contact surface." For example, pressure distribution information can be "input" into such a trained model, and at least one of the position, type, mass, and volume of the food can be directly "output." When the position of the food is output, the coordinates of the food's position can be output. In other embodiments, these inputs and / or outputs can be performed via images. That is, an image with pressure distribution information or the like added can be "input" into the trained model, and at least one of the position, type, mass, and volume of the food can be "output." Alternatively, only an image containing food without pressure distribution information or the like added can be "input" into the trained model, and at least one of the position, type, mass, and volume of the food can be "output." An example of a method for generating such a trained model is described below. First, predetermined characteristic distribution data is collected as training data (examples of "characteristic distributions" include pressure distribution, contact distribution, shading distribution, volume distribution, etc.). Next, the location, type, and / or mass of the food is annotated according to the property distribution. Such annotated property distribution is also used as another training data. A trained model can be generated by learning using this training data (at this time, parameters such as weights are set and updated). The information on the location, type, mass, etc. of the food obtained as described above can be used for the annotation work. Furthermore, there are no particular limitations on "machine learning" as long as it can perform the input and output described above, and various methods such as deep learning and supervised learning can be used.

[0052] The trained model may be generated by an external device or may be generated in the server 10. Note that the server 10 may be provided with a means for generating the trained model. The trained model may function by the control unit 12 executing a program stored in the storage unit 42, or may function by a device other than the server 10 executing a predetermined program. In the latter case, the trained model generated by the other device may be transmitted to the server 10 and stored in the storage unit 42 of the server 10.

[0053] (Data providing means 36) The data assigning means 36 assigns (annotates) at least one of the following data to the image data of a dish: the location of the dish, the type of dish, and the mass of the dish. That is, the data assigning means 36 may assign at least the location of the dish to the image data of a dish, or the type and / or mass of the dish to the image data of a dish. Furthermore, the data assigning means 36 may also assign the predicted volume of the dish to the image data of a dish. The term "assigning (annotating)" data to an image is not particularly limited; it is sufficient that an image (more specifically, one placement member 50) is associated with information such as the location, type, and mass of the dish (any information or combination that can be associated with the image; hereinafter, also referred to as "annotation information") (see FIGS. 4 to 6). The storage method is not particularly limited; it is sufficient that when an image is acquired from such a dataset, the annotation information (name of the dish and location information) of the image can be acquired in some form (text, two-dimensional array, etc.). FIG. 4 shows an example in which the type of food (food name), position (X position coordinate, Y position coordinate), and mass are annotated on an image, but only one piece of annotation information may be used. Position information may also be expressed as point information (such as the center of gravity). FIG. 5 shows an example in which the type of food (menu information) and a two-dimensional array of pressure distribution are annotated on an image as annotation information (dish information corresponding to the type of food is also annotated). The two-dimensional array of pressure distribution may be expressed using a mapping transformation or the like.

[0054] The image data of dishes to which information such as the position of the dishes has been added by the data adding means 36 can be used as training data for the trained model described above. Additionally, such image data can be used as source data for obtaining a new data set (such as data augmentation). For example, position information and pressure distribution can be estimated from this image data using a predetermined trained model, and the estimation results can be used as annotations. The same applies to the composite image generated by the image synthesis means 40 described below.

[0055] The data assigning means 36 may also assign preset menu information data to the data of the image of the dish. Here, the "preset menu information" is different from the information about the dish identified and acquired by the dish position acquiring means 24, the dish type identifying means 26, the dish mass identifying means 28, etc. In other words, in this case, the data assigning means 36 can annotate (link) the menu information, such as the dish name, that is predicted to be included in the image of the dish to the image data without using the placement member 50.

[0056] Figure 6 shows an example where menu information has been roughly annotated as candidate dish names. In this example, the annotation information is the names of dishes (menu information) such as "croquettes," "rice," and "miso soup." Furthermore, in addition to the names of dishes, menu information may also include information on the types and amounts of ingredients and seasonings, such as recipes (100g potatoes, 1 teaspoon soy sauce), and by using this information, it is possible to annotate more detailed information.

[0057] (Conversion means 38, image synthesis means 40) The conversion means 38 converts the two-dimensional image (left side of FIG. 7) generated from the pressure distribution information acquired by the pressure distribution information acquisition means 16 so that it matches the shape of the support member 50 in the image accepted by the image accepting means 14 (right side of FIG. 7). The image synthesis means 40 generates a synthesized image by superimposing the converted pressure distribution image on the image accepted by the image accepting means 14 (right side of FIG. 7).

[0058] Note that instead of a two-dimensional image generated from pressure distribution information, the conversion means 38 may convert a two-dimensional image generated from, for example, contact distribution information or shading distribution information acquired by the contact distribution information acquisition means 18 or the shading distribution information acquisition means 20 so that the converted image matches the shape of the support member 50 in the image received by the image receiving means 14. In this case, the image synthesis means 40 may generate a synthesized image by superimposing the converted contact distribution or shading distribution image on the image received by the image receiving means 14. The information about the food that is added to the captured image of the food is not limited to pressure distribution, contact distribution, or shading distribution, but may also include the mass or volume of the food. Furthermore, this information, including the position of the food (plate), may be updated and acquired over time.

[0059] The conversion method used by the conversion means 38 is not particularly limited, and may be, for example, projective transformation. Projective transformation can convert a rectangle in any space into a rectangle in another space (see FIG. 7). A characteristic of projective transformation is that it can convert into a rectangle in another space while maintaining the coordinate positional relationship on the same plane as the original rectangle. In other words, projective transformation makes it possible to deform and fit an image (or its shape) to fit another shape (frame). In an exemplary embodiment, the outline of the support member 50 is first detected from the food photo, and then projective transformation is used to reflect (combine) the "pressure information obtained from the support member 50" into the "food photo obtained by the imaging device 60." The composition method used by the image composition means 40 is also not particularly limited.

[0060] <Administrative terminal 46, staff terminal 48> The management terminal 46 and the staff terminal 48 can be configured from industrial computers, personal computers, tablet terminals, mobile terminals, etc., and are not particularly limited. In this specification, an "administrator" who operates the management terminal 46 is a person (including a corporation) who manages the server 10, and may manage the server 10 via the management terminal 46. A "staff member" who operates the staff terminal 48 is a person who manages the server 10 via the staff terminal 48 under the direction and supervision of the manager. Note that the manager or staff member may operate the server 10 directly.

[0061] The management terminal 46 and the staff terminal 48 each have a control unit that controls each unit (display unit, etc.) of the corresponding terminal. The manager or staff can manage the server 10 by operating an operation unit such as a keyboard while looking at the display unit, etc. The management terminal 46 and the staff terminal 48 can send and receive predetermined instructions and information to the server 10 via the communication unit, and can store predetermined data in the memory unit.

[0062] <Placement member 50> A plate with food on it is placed on the receiving member 50. In the example shown in FIG. 1, the receiving member 50 has a detection unit 52 and a communication unit 54. Examples of the receiving member 50 include trays, desks, placemats, table mats, serving shelves, and the plate placing unit of a serving robot, and are not limited to any object with a base surface on which a plate can be placed. In particular, tray-like receiving members 50 are easily portable, making them well suited to detecting the weight of food and can be used for serving, transporting, eating, and clearing meals. Since most facilities and hospitals use meal trays, the receiving member 50 can be introduced without changing on-site operations.

[0063] There are no limitations on the number or types of plates that can be placed on them. In the present application, containers for holding beverages such as water are also included in the term "plate." As shown in FIG. 3, it is preferable to know in advance information such as the type (size), weight, volume, and shape of the surface on which the plate on which food will be placed is known, and it is even more preferable to know in advance information on the type of food (menu, serving) to be placed on such plates. For example, if toast is placed on plate type A, soup on plate type B, and salad on plate type C (when the plate information and food information correspond), then if the type of plate is known, the type of food can be identified, and if the position of the plate is known, the position of the food can be identified.

[0064] (Detection unit 52) The detection unit 52 detects information on the pressure distribution applied to the support member 50. For example, when multiple dishes (plates) are placed on the base of the support member 50 as shown on the right side of FIG. 7, two-dimensional array data of pressure intensity as shown on the left side of FIG. 7 is obtained. The detection unit 52 (support member 50) may acquire the position of the plates on the support member 50 and the shape of the contact surface from the pressure distribution. Furthermore, in addition to or instead of the pressure distribution, the detection unit 52 may detect information on the contact distribution of members in contact with the support member 50 and information on the shading distribution of members shading the support member 50. The shape of the contact surface can also be identified from the information on the contact distribution and the shading distribution. Specific examples of such a detection unit 52 include a pressure sensor, a tactile sensor, a contact sensor, and an optical sensor.

[0065] It is particularly preferable that the detection unit 52 be a grid-like arrangement of small tactile sensors, pressure sensors, or other sensors embedded in the support member 50 (a large number of sensors are embedded in a flat pattern; see the bottom right of Figure 8). With such a sensor array type detection unit 52, each sensor can detect contact and pressure, and when an object comes into contact, the shape of the contact surface and the weight of the object can be determined. As a result, the position, contact surface shape, weight, etc. of a plate can be detected anywhere on the base surface of the support member 50, and this information can be obtained regardless of where the plate is placed on the base of the support member 50 during or after a meal.

[0066] The detection unit 52 may also include a sensor that detects a wireless tag attached to a plate. For example, the distance between the sensor and the wireless tag may be detected based on the strength of the radio wave using the received signal strength indicator (RSSI), and XY coordinates indicating the position of the plate on the support member 50 (or in the food image) may be calculated based on this distance. In the case of a sensor array type detection unit 52, the position of each wireless tag can be regarded as the position of each plate. Note that, in this application, the term "wireless tag" is not particularly limited (it may also be called an IC tag, RF tag, RFID tag, or non-contact tag), and may be either a passive tag or an active tag. That is, it may include an antenna that wirelessly communicates with the sensor attached to the support member 50, an IC chip that records information, and a battery for communication. The method of attaching the wireless tag to the plate is also not limited. The wireless tag may be embedded in the plate (built-in), attached to the side of the plate, or attached with a restraining member (such as a hook, string, or rubber band).

[0067] The sensor array detector 52 enables time-series monitoring of an individual's food intake. Detailed dietary monitoring is becoming increasingly necessary in the context of preventive medicine, particularly personalized nutrition. Currently, the sensor array detector 52 is ideal for situations where accurate understanding of diet and nutrition is highly required, such as in elderly care facilities and hospitals, to record food intake and address frailty and malnutrition. The conventional technology shown in Figure 8 measures the weight of plates and food in several predetermined areas (approximately 4–6 areas). This means that the areas where weight can be measured on this meal tray are predetermined, and one plate must be placed in each of the predetermined areas, resulting in limited flexibility. Furthermore, the plate must be kept within the corresponding area even during meal consumption, which is time-consuming for the user. In contrast, the sensor array detector 52 offers a high degree of flexibility, as described above. Furthermore, detecting the state of an individual's eating behavior over time is beneficial for monitoring eating behavior. For example, if it is detected that the pressure of the chopsticks has disappeared from the support member 50 and the pressure of plate A has increased by the weight of the chopsticks, it can be determined that the chopsticks have been placed on plate A, and if hand pressure is applied to the support member 50 for several minutes but the plate pressure information is detected as unchanged, it can be determined that the person has stopped eating but has not left their seat. In other words, the sensor array type detection unit 52 can monitor not only the actual amount and speed of food intake but also eating behavior.

[0068] (Communications Department 54) The communication unit 54 connects the placement member 50 wirelessly or with a wire so that the placement member 50 can communicate with an external device (such as the server 10). The placement member 50 can transmit (including uploading) detected data (such as pressure distribution) to the external device via the communication unit 54. The communication unit 54 may transmit data every time a value detected by the detection unit 52 changes, or may transmit data continuously. When transmitting data, it is preferable to also transmit the transmission time. This is because by comparing data before and after a meal, it is possible to obtain data such as eating speed, food preferences, appetite, and physical condition. On the server 10 side, it is also preferable that the plate position acquisition means 22, etc., acquire (store) the acquisition time each time they acquire various types of data.

[0069] <Imaging device 60> The imaging device 60 captures an image of a plate or dish placed on the placement member 50. In the example shown in Fig. 1, the imaging device 60 has a control unit 62, an imaging unit 64, and a communication unit 66. Specifically, the imaging unit 64 captures an image of the plate or dish placed on the placement member 50.

[0070] (control unit 62) The control unit 62 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and controls the operation of the imaging device 60.

[0071] (Image capture unit 64) Examples of the imaging unit 64 include so-called smartphone cameras and fixed cameras (IoT cameras), but from the perspective of privacy during meals, it may be preferable to use a surveillance camera or the like. There are no particular limitations on the imaging range of the imaging unit 64; multiple mount members 50 may be included in one image (see FIG. 4), or multiple imaging units 64 may capture the outline of one mount member 50 (the four vertices of a rectangle). With the widespread use of smartphones, people often take photos of food before eating with cameras, but in most cases, these are taken from directly above at close range. In contrast, there are no limitations on the imaging method used by the imaging unit 64; images may be taken from various distances and angles.

[0072] The imaging unit 64 may also be provided with a depth sensor 64a that can estimate the volume of food on each plate. When capturing an image of food, the depth sensor 64a can also directly obtain information about the volume of the food.

[0073] (Communications Department 66) The communication unit 66 connects the imaging device 60 to an external device (such as the server 10) wirelessly or via a wired connection so that the imaging device 60 can communicate with the external device. The imaging device 60 can transmit (including upload) information (including images and videos) to the external device via the communication unit 66. When transmitting a video, it is preferable to also transmit the transmission time at the same time, as this allows each image constituting the video to correspond to the pressure information from the mounting member 50.

[0074] [Information processing method 1] Next, information processing method 1 in the information processing system 1 described above will be described with reference to FIG. 9. Information processing method 1 is an example in which data on an image of a dish is provided with information about the dish (the position and mass of the dish). In the following description, components with the same reference numerals are the same as those described above, and redundant description will be omitted as appropriate. Note that the processing described below is performed by executing a program stored in storage unit 42, but the information processing method according to the present disclosure is not limited to this. It is also assumed that the relationship between the type of dish to be placed on placement member 50 and the shape of the contact surface of the dish, as well as the mass of the dish, is set in advance.

[0075] First, the control unit 12 (image receiving means 14) receives, from the imaging device 60, image data of the placement member 50 captured by the imaging device 60 and the food placed on the plate placed on the placement member 50 (step S10).

[0076] Next, the control unit 12 (pressure distribution information acquiring means 16) receives information on the pressure distribution applied to the placement member 50 from the placement member 50 (step S20).

[0077] Next, the control unit 12 (food mass determination means 28) determines the shape of the contact surface of the plate placed on the support member 50 and the mass of the food to be placed on the plate from the pressure distribution information acquired by the support member 50 (step S30).

[0078] Next, the control unit 12 (food type identification means 26) identifies the type of the identified dish from the shape of the contact surface of the dish (step S40).

[0079] Next, the control unit 12 (dish type identification means 26) identifies the type of dish to be placed on the plate from the information on the identified plate type (step S50).

[0080] Next, the control unit 12 (data providing means 36) provides data on the weight of the dish and information on the type of dish to the data on the image of the dish received in step S10 (step S60; for example, FIGS. 4 to 6).

[0081] The information processing method described above is an example, and the processing flow is not limited to the above. For example, the step of receiving image data related to food (step S10) may be executed after step S50 (as long as it is executed before step S60).

[0082] In another embodiment, in step S20, instead of pressure distribution information, information on the contact distribution of dishes or the like that are in contact with the support member 50 or information on the shading distribution of dishes or the like that are shading the support member 50 may be received, and the shape of the contact surface of the dish is identified based on either of these characteristic distributions (the mass of the food is not obtained).

[0083] In yet another embodiment, the control unit 12 (dish position acquisition means 22) acquires information about the type and position of the dish (on the placement member 50) from the shape of the identified plate's contact surface, the control unit 12 (food position acquisition means 24) acquires food position information from the acquired dish position information based on pre-set information about the food to be placed on the plate, and the control unit 12 (data assignment means 36) can assign data about the food position information to the food image data received in step S10. Alternatively, the control unit 12 (dish position acquisition means 22) may acquire information about the dish's position on the placement member 50 from information about any of the characteristic distributions. These processes can be efficiently implemented.

[0084] In yet another embodiment, if a correspondence between the type of dish and the 3D structural data (shape) of the dish is set in advance, the control unit 12 (dish shape identification means 30) identifies the shape of the dish corresponding to the type of dish identified in step S40, the control unit 12 (food volume prediction means 32) predicts the volume of the food placed on this dish from the identified shape of the dish, and the control unit 12 (data assignment means 36) can assign data on the predicted food volume information to the data of the image of the food received in step S10.

[0085] [Information processing method 2] Next, information processing method 2 in the information processing system 1 will be described with reference to Fig. 10. Information processing method 2 is an example in which pressure distribution information received from the mounting member 50 is projectively transformed and superimposed on an image received from the imaging device 60 to generate a composite image (see Fig. 7). Steps S110 to S120 are the same as steps S10 to S20 in information processing method 1, and therefore description thereof will be omitted.

[0086] After step S120, the control unit 12 (conversion means 38) performs projective transformation on the two-dimensional image (left side of Figure 7) generated from the pressure distribution information received from the support member 50 so that it matches the shape of the support member 50 in the image received from the imaging device 60 (step S130).

[0087] Next, the control unit 12 (image synthesis means 40) generates a synthetic image by superimposing the pressure distribution image (left side of Figure 7) that has undergone projective transformation on the image received from the imaging device 60 in step S110 (step S140, right side of Figure 7).

[0088] The information processing method described above is an example, and the processing flow is not limited to the above. For example, the step of receiving image data relating to food from the imaging device 60 (step S110) may be performed after step S130 (as long as it is performed before step S140). In another embodiment, a composite image may be generated by performing projective transformation on the data of information on the mass and type of food obtained as in information processing method 1 so that the data matches the shape of the placement member 50 in the image received from the imaging device 60.

[0089] The program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure configured as described above include image receiving means 14 and data assigning means 36. Image receiving means 14 receives data on an image of a dish captured by imaging device 60 from imaging device 60, and data assigning means 36 assigns at least one of information on the position of the dish, information on the type of dish, and information on the weight of the dish to the image data of the dish.

[0090] As mentioned above, the concept of individualized optimization of diet and nutrition (individually optimized preventive medicine) has been advocated in recent years. However, in epidemiological studies, food records have traditionally been recorded in text format. Since the widespread use of smartphones, epidemiological studies have increasingly used pre-meal photos of food taken with a camera and in-app AI to estimate the dish name and nutrient content. However, developing AI capable of estimating nutritional information with greater accuracy requires annotated food image datasets, including location, dish name, and weight information for each dish, as AI training data. Image annotation is typically performed manually, which requires enormous human resources. To address this issue, the program, information processing system 1, server (computer) 10, and information processing method disclosed herein efficiently add information about food (such as mass) to food image data. In other words, this solves the trade-off between improving annotation quality and increasing effort when building a food image dataset, thereby enabling efficient construction of high-quality annotated food image datasets. In addition, since food-related data (such as mass) can be obtained over time from images of food during a meal, for example, if a person shows a tendency to binge eat staple foods, it can be used to advise them that this is a contributing factor to poor postprandial blood sugar control. It can also identify tendencies toward picky eating and loss of appetite, which can be useful in nursing homes, hospitals, and other settings. Furthermore, the program, information processing system 1, server (computer) 10, and information processing method disclosed herein are compatible with IoT camera infrastructure and can also be used for dietary and nutritional management for ordinary people in homes and workplaces, where IoT cameras are expected to become widespread. With IoT camera infrastructure, food images captured from various distances and angles must be analyzed using AI, and current food analysis AI trained on food photo datasets captured directly from above using smartphones has limitations. According to this disclosure, it is also easy to build a food image dataset based on IoT camera infrastructure.

[0091] In detail, in the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure, the data assigning means 36 may assign data of preset menu information to data of an image relating to a dish. In a hospital facility or the like where menu information is predetermined, if the date and time of an image capture is known, it is possible to annotate the image with a candidate dish name, and annotation information can be obtained efficiently without using the placement member 50.

[0092] Furthermore, the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure may further include a plate position acquisition means 22 and a food position acquisition means 24. The plate position acquisition means 22 acquires information about the position of the plate on the support member 50 on which the plate carrying the food is placed, and the food position acquisition means 24 acquires information about the food position from the acquired information about the plate position based on preset information about the food to be placed on the plate. The plate position acquisition means 22 may also receive and acquire information about the plate position acquired by the support member 50 from a wireless tag attached to the plate from the support member 50. In this way, the position information of the food can be obtained from the position information of the plate obtained via the support member 50, and this position information of the food can be assigned to image data related to the food.

[0093] Furthermore, in the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure, the shape of the contact surface of the dish is set in advance, and the dish position acquisition means 22 receives information relating to the shape of the contact surface of the dish placed on the support member 50 from the support member 50, and can acquire information relating to the position of the dish on the support member 50 from the received information relating to the shape of the contact surface of the dish. When the "shape of the contact surface of the dish placed on the support member 50" is set in advance, knowing the shape of the contact surface of the dish also determines the type of corresponding dish and the position of that dish on the support member 50. The following are examples of ways to set in advance the "shape of the contact surface of the dish placed on the support member 50".

[0094] Specifically, in the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure, pressure distribution information acquisition means 16 may be further provided for acquiring information on the distribution of pressure applied to the support member 50 from the support member 50, in which case the dish position acquisition means 22 can identify the shape of the contact surface of the dish placed on the support member 50 from the pressure distribution information acquired by the pressure distribution information acquisition means 16. Also, contact distribution information acquisition means 18 may be further provided for acquiring information on the contact distribution of members in contact with the support member 50 from the support member 50, in which case the dish position acquisition means 22 can identify the shape of the contact surface of the dish placed on the support member 50 from the contact distribution information acquired from the contact distribution information acquisition means 18. Furthermore, a shading distribution information acquisition means 20 may be further provided which acquires, from the support member 50, information on the shading distribution of the member shading the support member 50, and in this case, the plate position acquisition means 22 can identify the shape of the contact surface of the plate placed on the support member 50 from the shading distribution information acquired from the shading distribution information acquisition means 20. Such pressure distribution information acquisition means 16, contact distribution information acquisition means 18, and shading distribution information acquisition means 20 may be provided alone or in any combination.

[0095] Furthermore, in the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure, a relationship between the type of plate and the shape of the plate's contact surface may be preset, and a relationship between the type of plate and the type of food to be placed on the plate may also be preset. A food type identification means 26 as described below may also be provided. This food type identification means 26 receives information regarding the shape of the contact surface of the plate placed on the support member 50 from the support member 50, identifies the type of plate from the shape of the plate's contact surface, and identifies the type of food to be placed on the plate from the identified information about the dish type. In this way, this food type information can be added to image data related to the food. In this case, if the pressure distribution information acquisition means 16 acquires information about the pressure distribution applied to the support member 50 from the support member 50, the dish position acquisition means 22 can identify the shape of the contact surface of the plate to be placed on the support member 50 from the pressure distribution information acquired by the pressure distribution information acquisition means 16. As a result, as mentioned above, the plate position acquisition means 22 can also acquire information regarding the position of the plate on the support member 50 from information regarding the shape of the plate's contact surface, and this position information of the dish can also be added to the image data related to the dish.

[0096] Furthermore, the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure may further include food mass identification means 28 that identifies the mass of the food placed on the plate from the pressure distribution information acquired by pressure distribution information acquisition means 16. In this way, data assignment means 36 can assign this food mass information to data on the image of the food.

[0097] Furthermore, the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure may further include a plate shape identification means 30 and a food volume prediction means 32. The plate shape identification means 30 identifies the type of dish from an image of the dish captured by the imaging device 60 and identifies the shape of the dish corresponding to the identified type of dish, and the food volume prediction means 32 predicts the volume of the food placed on the plate from the identified shape of the dish. In this way, the data assignment means 36 can also assign data on the predicted food volume to the data of the image of the dish.

[0098] Furthermore, the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure may further include pressure distribution information acquisition means 16 and cooking content prediction means 34. The pressure distribution information acquisition means 16 acquires information on the pressure distribution applied to the placement member 50 from the placement member 50, and the cooking content prediction means 34 predicts at least one of the position, type, and mass of the food from the pressure distribution information acquired from the pressure distribution information acquisition means 16 and the image of the food captured by the imaging device 60. Data on the information about the food predicted in this way is added to data on the image of the food.

[0099] Furthermore, the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure may further include conversion means 38 and image synthesis means 40. When the pressure distribution information acquisition means 16 acquires information about the pressure distribution applied to the support member 50 from the support member 50 on which a plate with food placed thereon is placed, the conversion means 38 performs conversion (such as projective transformation) on the two-dimensional image generated from the pressure distribution information acquired by the pressure distribution information acquisition means 16 so that it matches the shape of the support member 50 in the image accepted by the image accepting means 14, and the image synthesis means 40 generates a synthesized image by overlaying the converted pressure distribution image on the image accepted by the image accepting means 14. In this way, data about the image of the food and data about the food can be used.

[0100] Moreover, one embodiment of the information processing system 1 includes a server (computer) 10 and a support member 50 on which a plate carrying food is to be placed, the support member 50 having a detection unit 52 that detects information on the pressure distribution applied to the support member 50, and the server (computer) 10 executes a program to function as an image receiving means 14, a pressure distribution information acquiring means 16, a plate position acquiring means 22, a food position acquiring means 24, and a data providing means 36. Here, the image receiving means 14 receives image data of the support member 50 captured by the imaging device 60 and the food placed on the plate placed on the support member 50. The pressure distribution information acquiring means 16 acquires information on the pressure distribution applied to the support member 50 detected by the detection unit 52 from the support member 50. The plate position acquisition means 22 identifies the shape of the contact surface of the plate placed on the support member 50 from the pressure distribution information acquired by the pressure distribution information acquisition means 16, and acquires information about the position of the plate on the support member 50 from the information about the identified shape of the contact surface of the plate. The food position acquisition means 24 acquires information about the position of the dish from the acquired information about the plate position based on preset information about the food to be placed on the plate. The data assignment means 36 assigns at least data about the position of the food to data about the image of the food.

[0101] Furthermore, in the program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure, a relationship between the type of plate and the shape of the plate's contact surface may be preset, and a relationship between the type of plate and the type of food placed on the plate may also be preset, and a food type identification means 26 as described below may be further provided. This food type identification means 26 identifies the type of plate from the identified shape of the plate's contact surface, and identifies the type of food placed on the plate from information on the identified food type. In this way, the data assignment means 36 can also assign data on the identified food type to data on an image of food.

[0102] Moreover, one embodiment of the information processing system 1 includes a server (computer) 10 and a plate receiving member 50 on which a plate carrying food is to be placed, the plate receiving member 50 having a detection unit 52 that detects information on the pressure distribution applied to the plate receiving member 50, and the server (computer) 10 functions as pressure distribution information acquiring means 16 and plate position acquiring means 22 by executing a program. Here, the pressure distribution information acquiring means 16 acquires information on the pressure distribution applied to the plate receiving member 50, detected by the detection unit 52, from the plate receiving member 50. The plate position acquiring means 22 acquires information on the position of the plate on the plate receiving member 50 from the pressure distribution information acquired by the pressure distribution information acquiring means 16.

[0103] The program, information processing system 1, server (computer) 10, and information processing method according to the present disclosure are not limited to the above-described aspects and combinations, and various modifications can be made.

[0104] For example, the type and position of food can be identified using something other than the placement member 50. As an example, by providing a wireless tag (IC chip) as described above on a plate and presetting a correspondence between this tag and the type of food placed on the corresponding plate, the wireless tag can be detected by the imaging device 60 or the like, and the type and position of food can be identified based on information regarding the size, position, etc. of the plate on which the wireless tag is provided.

[0105] Furthermore, in the above description, the case has been described in which the plate position acquisition means 22 receives information about the position of the plate acquired by the placement member 50 from the wireless tag, but if the relationship between the wireless tag attached to the plate and information about the plate (type, shape, etc.), information about the food to be placed on the plate (position, type, mass, etc.), etc. is set in advance, the information associated with the wireless tag detected by the placement member 50 via a sensor can also be used by the food position acquisition means 24, food type identification means 26, dish shape identification means 30, food volume prediction means 32, data assignment means 36, etc. Note that the means for detecting the information associated with the wireless tag does not always need to be provided on the placement member 50, but may be provided near the placement member 50 or in the above-mentioned imaging device 60 (surveillance camera, etc.).

[0106] In addition, the control unit 62 of the imaging device 60 may function as the above-mentioned plate position acquisition means 22, food position acquisition means 24, food type identification means 26, food mass identification means 28, plate shape identification means 30, food volume prediction means 32, food content prediction means 34, etc., and may cooperate with the control unit 12 of the server (computer) 10. [Explanation of symbols]

[0107] 1. Information Processing Systems 10 Servers (computers) 12 Control Unit 14 Image reception method 16 Pressure distribution information acquisition means 18 Contact distribution information acquisition means 20 Shading distribution information acquisition means 22 Plate position acquisition means 24 Food position acquisition means 26 Food type identification method 28 Food quantity identification means 30 Dish shape identification means 32 Food volume prediction method 34 Cooking Content Prediction Method 36 Data provision means 38 Conversion Methods 40 Image synthesis means 42 Storage section 44 Communications Department 46 Management terminal 48 Staff terminal 50 Mounting member 52 Detection unit 54 Communications Department 60 Imaging device 62 Control unit 12 64 Control Unit 12 64a Depth sensor 66 Communications Department

Claims

1. A program that causes a computer to function as an image receiving means and a data providing means, the image receiving means receives, from the imaging device, data of an image relating to food captured by the imaging device; The data assigning means assigns at least one of information on the position of the food, information on the type of food, and information on the weight of the food to data on an image of the food.

2. 2. The program according to claim 1, wherein the data providing means provides data of preset menu information to data of an image relating to food.

3. The computer further functions as a plate position acquisition means and a food position acquisition means, the plate position acquisition means acquires information about the position of the plate on a support member on which the plate carrying food is placed, 2. The program according to claim 1, wherein the food position acquisition means acquires food position information from the acquired plate position information based on preset information about the food to be placed on the plate.

4. 4. The program according to claim 3, wherein the plate position acquisition means receives, from the plate receiving member, information about the plate position acquired by the plate receiving member from a wireless tag attached to the plate.

5. The shape of the plate's contact surface is preset, The program described in claim 3, wherein the plate position acquisition means receives information regarding the shape of the contact surface of the plate placed on the support member from the support member, and acquires information regarding the position of the plate on the support member from the received information regarding the shape of the contact surface of the plate.

6. causing the computer to further function as pressure distribution information acquisition means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; 5. The program according to claim 4, wherein the plate position acquisition means identifies the shape of the contact surface of the plate placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means.

7. causing the computer to further function as contact distribution information acquisition means, the contact distribution information acquiring means acquires, from the support member, information on the contact distribution of members in contact with the support member; 5. The program according to claim 4, wherein the plate position acquiring means identifies the shape of the contact surface of the plate placed on the support member from the contact distribution information acquired from the contact distribution information acquiring means.

8. causing the computer to further function as a light-shading distribution information acquisition unit; the light-blocking distribution information acquisition means acquires, from the support member, information on the light-blocking distribution of a member that is blocking the support member; 5. The program according to claim 4, wherein the dish position acquisition means identifies the shape of the contact surface of the dish placed on the support member from the information on the light-blocking distribution acquired from the light-blocking distribution information acquisition means.

9. The computer further functions as a food type identification means, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, The program described in claim 1, wherein the food type identification means receives information from the support member regarding the shape of the contact surface of the plate to be placed on the support member, identifies the type of plate from the shape of the contact surface of the plate, and identifies the type of food to be placed on the plate from the information on the identified type of plate.

10. The computer is further caused to function as a pressure distribution information acquisition means and a plate position acquisition means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; 10. The program according to claim 9, wherein the plate position acquisition means identifies the shape of the contact surface of the plate placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means.

11. The computer functions as a food mass determination means, The program according to claim 10 , wherein the food mass specifying means specifies the mass of the food placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means.

12. The computer is further caused to function as a dish shape specifying means and a dish volume predicting means, the dish shape identification means identifies the type of dish from the image of the food captured by the imaging device, and identifies the shape of the dish corresponding to the identified type of dish; the food volume prediction means predicts the volume of food placed on the plate from the identified shape of the plate; 2. The program according to claim 1, wherein said data providing means also provides data on predicted dish volume information to the data on the image of the dish.

13. The computer is further configured to function as a pressure distribution information acquisition means and a cooking content prediction means, the pressure distribution information acquiring means acquires information on the pressure distribution applied to the support member from the support member; The program of claim 1, wherein the cooking content prediction means predicts at least one of the position of the food, the type of food, and the mass of the food from the pressure distribution information acquired from the pressure distribution information acquisition means and the image of the food captured by the imaging device.

14. causing the computer to further function as a pressure distribution information acquiring means, a converting means, and an image synthesizing means; the pressure distribution information acquiring means acquires information on the pressure distribution applied to a support member on which a plate with food placed thereon is placed, the converting means converts the two-dimensional image generated from the pressure distribution information acquired by the pressure distribution information acquiring means so that the image matches the shape of the support member in the image accepted by the image accepting means; 2. The program according to claim 1, wherein the image synthesis means generates a synthesized image by superimposing the converted pressure distribution image on the image accepted by the image acceptance means.

15. A computer that functions as an image receiving means and a data providing means by executing a program, the image receiving means receives, from the imaging device, data of an image relating to food captured by the imaging device; The data assigning means assigns at least one of information on the position of the food, information on the type of food, and information on the weight of the food to data on the image of the food, a computer.

16. A computer, a support member on which a plate carrying food is placed; A system comprising: the support member has a detection unit that detects information on a pressure distribution applied to the support member, The computer executes a program to function as an image receiving means, a pressure distribution information acquiring means, a plate position acquiring means, a food position acquiring means, and a data assigning means, the image receiving means receives, from the imaging device, image data of the placement member and the food placed on a plate placed on the placement member, the image data being captured by the imaging device; the pressure distribution information acquisition means acquires, from the support member, information on the pressure distribution applied to the support member detected by the detection unit; the dish position acquisition means identifies the shape of the contact surface of the dish placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means, and acquires information about the position of the dish on the support member from the information about the identified shape of the contact surface of the dish; the food position acquisition means acquires food position information from the acquired dish position information based on preset information about the food to be placed on the plate; The data providing means provides at least data on the location of the food to data on an image of the food.

17. The computer further functions as a cuisine type identification means by executing the program, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, the food type identification means identifies the type of dish from the shape of the contact surface of the identified dish, and identifies the type of food to be placed on the dish from information on the identified type of dish; 17. The system according to claim 16, wherein said data providing means also provides data on the type of food to data on an image relating to food.

18. The computer further functions as a food mass determination means by executing the program, the food mass specifying means specifies the mass of the food to be placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means; 17. The system according to claim 16, wherein the data providing means also provides data on the mass of the food to the data on the image relating to the food.

19. A computer, a support member on which a plate carrying food is placed; A system comprising: the support member has a detection unit that detects information on a pressure distribution applied to the support member, The computer executes a program to function as a pressure distribution information acquisition means and a plate position acquisition means, the pressure distribution information acquisition means acquires, from the support member, information on the pressure distribution applied to the support member detected by the detection unit; The plate position acquisition means acquires information about the position of the plate on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means.

20. The system described in claim 19, wherein the plate position acquisition means identifies the shape of the contact surface of the plate placed on the support member from the pressure distribution information acquired by the pressure distribution information acquisition means, and acquires information regarding the position of the plate on the support member from the information regarding the identified shape of the contact surface of the plate.

21. The computer further functions as a cuisine type identification means by executing the program, A relationship between the type of plate and the shape of the plate's contact surface is preset, and a relationship between the type of plate and the type of food to be placed on the plate is preset, 20. The system according to claim 19, wherein the food type identification means identifies the type of dish from the shape of the contact surface of the identified dish, and identifies the type of food to be placed on the dish from information on the identified type of dish.

22. The computer further functions as a food mass determination means by executing the program, The system according to claim 20 , wherein the food mass specifying means specifies the mass of the food placed on the plate from the pressure distribution information acquired by the pressure distribution information acquiring means.

23. An information processing method performed by a computer having a control unit, the control unit receives, from an imaging device, data of an image of food captured by the imaging device; The information processing method, wherein the control unit adds at least one of information on the position of the food, information on the type of food, and information on the mass of the food to data on an image of the food.

Images