Information processing device, display control method, and display control program

Abstract

In order to display information pertaining to a food product in an overlapping manner at a suitable position, this information processing device (1) comprises: a detecting unit (101) that detects a prescribed detection target from an image in which at least a portion of the field of view of a user is imaged and in which the food product is shown; an overlapping region setting unit (102) that sets an overlapping region for information pertaining to the food product in the image on the basis of the detected target; and a display control unit (103) that displays the information in an overlapping manner in the overlapping region.

Description

[Technical Field]

[0001] This relates to AR (Augmented Reality) technology that displays information superimposed on an image that captures at least a part of the user's field of vision. [Background technology]

[0002] The use of AR technology in various applications is being considered. For example, Patent Document 1 below discloses an HMD (Head Mounted Display) that uses AR technology to manage calorie intake. The HMD in Patent Document 1 displays a blue filter superimposed on food when the calorie intake limit of the user wearing the HMD falls below a certain value. Through this process, the HMD in Patent Document 1 reduces the user's appetite when the calorie intake limit falls below a certain value, thereby controlling the user's calorie intake. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-042401 Summary of the Invention [Problem to be solved by the invention]

[0004] The blue filter superimposed on food in the HMD described in Patent Document 1 is intended to reduce the user's appetite, so the display position of the blue filter only needs to roughly match the position of the food displayed on the HMD. For this reason, Patent Document 1 does not specifically describe a configuration for superimposing information about food in an appropriate position.

[0005] One aspect of the present invention has been made in consideration of the above points, and one object thereof is to provide a technology that enables information about food to be superimposed and displayed at an appropriate position. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention comprises a detection means for detecting a predetermined detection target from an image capturing at least a portion of a user's field of vision, the image including food, a superimposition area setting means for setting a superimposition area for information related to the food in the image based on the detection target, and a display control means for superimposing the information on the superimposition area.

[0007] A display control method according to one aspect of the present invention includes at least one processor detecting a predetermined detection object from an image capturing at least a portion of a user's field of view, the image including food; setting a superimposition area for information related to the food in the image based on the detection object; and superimposing the information on the superimposition area.

[0008] A display control program according to one aspect of the present invention causes a computer to function as a detection means for detecting a predetermined detection target from an image capturing at least a portion of a user's field of vision, the image including food, an overlay area setting means for setting an overlay area for information related to the food in the image based on the detection target, and a display control means for superimposing and displaying the information in the overlay area. [Effects of the Invention]

[0009] According to one aspect of the present invention, information about food can be superimposed and displayed at an appropriate position. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of a display control method according to the first exemplary embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of use of an information processing device according to a second exemplary embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing an example of a main configuration of the information processing device. [Figure 5] 10A and 10B are diagrams illustrating an example of setting an overlapping area by the information processing device. [Figure 6] 10A and 10B are diagrams illustrating an example of generation of a superimposed image by the information processing device. [Figure 7] FIG. 10 is a flowchart showing the flow of a display control method according to a second exemplary embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing an example in which a plurality of food regions are set within the overlapping region and displayed in overlapping fashion. [Figure 9] FIG. 10 is a diagram showing an example in which a superimposed image is displayed with chopsticks held by a user as the detection target. [Figure 10] 10A and 10B are diagrams illustrating an example of setting an overlapping region in which a fork is to be detected, and an example of setting an overlapping region in which a user's hand holding the fork is to be detected. [Figure 11] FIG. 1 is a diagram illustrating an example of a computer that executes instructions of a program, which is software that realizes the functions of each device according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0012] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes a detection unit (detection means) 11, a superimposition area setting unit (superimposition area setting means) 12, and a display control unit (display control means) 13.

[0013] The detection unit 11 detects a predetermined detection target from an image that captures at least a part of the user's field of view and that includes food.

[0014] The superimposition area setting unit 12 sets a superimposition area for the information related to the food in the image based on the detection target detected by the detection unit 11.

[0015] The display control unit 13 causes the information to be superimposed and displayed in the superimposition area set by the superimposition area setting unit 12 .

[0016] As described above, the information processing device 1 according to this exemplary embodiment is configured to include a detection unit 11 that detects a predetermined detection target from an image capturing at least a portion of a user's field of view that includes food, a superimposition area setting unit 12 that sets a superimposition area for information related to the food in the image based on the detection target, and a display control unit 13 that superimposes and displays the information in the superimposition area. Therefore, the information processing device 1 according to this exemplary embodiment has the effect of being able to superimpose and display information related to food in an appropriate position.

[0017] (Display control program) The functions of the information processing device 1 described above can also be realized by a program. The display control program according to this exemplary embodiment causes a computer to function as a detection unit 11, a superimposition area setting unit 12, and a display control unit 13. This display control program has the effect of enabling information about food to be superimposed and displayed at an appropriate position.

[0018] (Flow of display control method) The flow of the display control method according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the display control method. Note that the execution entity of each step in this display control method may be a processor provided in the information processing device 1, or a processor provided in another device, or each step may be executed by a processor provided in a different device.

[0019] In S11, at least one processor detects a predetermined detection target from an image capturing at least a part of the user's field of view, the image including food.

[0020] In S12, at least one processor sets a superimposition area for the information related to the food in the image based on the detection target detected in S11.

[0021] In S13, at least one processor displays the information in a superimposed manner in the superimposition area set in S12.

[0022] As described above, the display control method according to this exemplary embodiment includes the steps of: detecting a predetermined detection target from an image capturing at least a portion of a user's field of view, the image capturing at least a portion of the user's field of view, setting a superimposition area for information related to the food in the image based on the detection target, and displaying the information superimposed on the superimposition area. This display control method has the effect of superimposing information related to the food in an appropriate position.

[0023] Exemplary Embodiment 2 (overview) An overview of an information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of how the information processing device 2 is used. As shown in the figure, the information processing device 2 is a head-mounted display that a user wears on their head. The information processing device 2 can display so-called AR (augmented reality) content.

[0024] For this reason, the information processing device 2 includes an image capturing unit 22 that captures at least a part of the visual field of the user wearing the information processing device 2, and a display unit 23 that displays the image captured by the image capturing unit 22. The information processing device 2 allows the user to experience augmented reality by superimposing AR content on the image captured by the image capturing unit 22.

[0025] When displaying AR content, the information processing device 2 detects a predetermined detection target from an image 31 that captures at least a portion of the user's field of view and that includes food. In the example of FIG. 3, the information processing device 2 detects a three-dimensional code OB1 attached to a table on which a lunch box is placed. Note that the above-mentioned "food" may be anything that the user can ingest, and for example, foods made from a single ingredient such as fruit, dishes made from multiple ingredients, supplements, medicines, beverages, etc. are also included in the category of "food."

[0026] Then, the information processing device 2 sets a superimposition area on the image captured by the photographing unit 22 based on the detected three-dimensional code OB1, and displays information about the food superimposed on the set superimposition area. Image 32 shown in Fig. 3 sets the area in image 31 where chicken breast (steamed chicken) is shown as the superimposition area, and displays an image of fried chicken superimposed on the superimposition area.

[0027] As a result, low-calorie chicken breast meat appears to a user wearing information processing device 2 as fried chicken meat, which has higher calories. This allows the user to reduce their calorie intake by actually eating chicken breast meat, while still feeling satisfied as if they had eaten fried chicken. Furthermore, information processing device 2 sets the superimposition area based on the detected three-dimensional code OB1, so that information about the food (an image of fried chicken in the example of FIG. 3) can be superimposed and displayed at an appropriate position.

[0028] The information processing device 2 is not limited to a head-mounted display, and may be, for example, a smartphone. When a smartphone is used as the information processing device 2, a fixture may be used to fix the smartphone display in front of the user's eyes. Furthermore, for example, a glasses-type or goggle-type display device may also be used as the information processing device 2.

[0029] (Configuration of information processing device 2) The configuration of the information processing device 2 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2. As shown in the figure, the information processing device 2 includes a control unit 20 that controls each unit of the information processing device 2, and a storage unit 21 that stores various data used by the information processing device 2.

[0030] The information processing device 2 also includes a photographing unit 22 for photographing an image, a display unit 23 for displaying the image, and an input unit 24 for receiving input operations to the information processing device 2. Note that the information processing device 2 may be configured to detect, for example, the line of sight of the user and operate using the direction and movement of the line of sight as input, in which case the input unit 24 may be omitted.

[0031] The control unit 20 also includes an image acquisition unit 201, a detection unit (detection means) 202, a superimposition area setting unit (superimposition area setting means) 203, a correction unit (correction means) 204, a superimposition image generation unit 205, and a display control unit (display control means) 206.

[0032] The image acquisition unit 201 acquires an image capturing at least a part of the user's visual field. The photographing unit 22 is disposed so as to capture an image in the line of sight of the user wearing the information processing device 2, and therefore the image captured by the photographing unit 22 is an image capturing at least a part of the user's visual field. Therefore, the image acquisition unit 201 acquires the image captured by the photographing unit 22. For example, the image acquisition unit 201 may acquire the image by extracting time-series frame images from a moving image captured by the photographing unit 22.

[0033] The detection unit 202 detects a predetermined detection target from the image acquired by the image acquisition unit 201. The predetermined detection target serves as a reference when setting a superimposition area for information related to food appearing in the image acquired by the image acquisition unit 201. For example, in the example of FIG. 3, the three-dimensional code OB1 is the predetermined detection target. Of course, the predetermined detection target is not limited to the three-dimensional code OB1 as long as it serves as a reference when setting the superimposition area.

[0034] Furthermore, the detection unit 202 detects color correction markers, which serve as a reference for color correction, from the images acquired by the image acquisition unit 201. Details of the color correction markers will be explained later in the section "Color Correction." Note that a processing block that detects the color correction markers may be provided separately from the detection unit 202.

[0035] The superimposition area setting unit 203 sets a superimposition area for information related to food in the image acquired by the image acquisition unit 201, based on the above-mentioned detection target detected by the detection unit 202. A specific method for setting the superimposition area will be described later in the section "Method for setting the superimposition area."

[0036] The correction unit 204 performs color correction on the image acquired by the image acquisition unit 201, using the above-described color correction markers detected by the detection unit 202. The color correction method will be described later in the section "Color Correction."

[0037] The superimposed image generating unit 205 generates a superimposed image to be superimposed on the image acquired by the image acquiring unit 201. The superimposed image may be an image containing information about food. For example, the superimposed image generating unit 205 may generate an image of a predetermined food as shown in the example of FIG. 3 as a superimposed image, or may generate a superimposed image containing attribute information about the food, such as the nutrients and calories contained in the food. The superimposed image generating unit 205 may also generate an image of an arbitrary character (e.g., a character preferred by the user) as a superimposed image. The character may be displayed as a still image or as a moving image. The character may also be a two-dimensional image or a three-dimensional image. For example, the superimposed image may be an image of a virtual character from an anime or manga, or an image of a real person (including a celebrity). The original image of the superimposed image (e.g., texture images of various foods or character images) may be stored in the storage unit 21 or the like. A method for generating a superimposed image will be described later in the section "Method for Generating a Superimposed Image."

[0038] The display control unit 206 superimposes and displays information related to food in the superimposition area set by the superimposition area setting unit 203. Specifically, the display control unit 206 superimposes and displays the information related to food by combining the superimposition image generated by the superimposition image generation unit 205 with the superimposition area set by the superimposition area setting unit 203 and displaying the combined image.

[0039] As described above, the information processing device 2 according to this exemplary embodiment includes a detection unit 202 that detects a predetermined detection target from an image capturing at least a portion of the user's field of view and that includes food, a superimposition area setting unit 203 that sets a superimposition area for information related to the food in the image based on the detection target, and a display control unit 206 that superimposes and displays the information in the superimposition area. Therefore, the information processing device 2 according to this exemplary embodiment has the effect of being able to superimpose and display information related to food in an appropriate position.

[0040] As described above, in the information processing device 2 according to this exemplary embodiment, the display control unit 206 may superimpose an image of a predetermined food on the superimposition area, thereby achieving the effect of increasing meal satisfaction in addition to the effect achieved by the information processing device 1 according to exemplary embodiment 1.

[0041] The predetermined food may be a different food from the food shown in the original image, or it may be the same food. When an image of the same food is displayed, the amount of food may be different from the amount of food shown in the original image. By superimposing an image showing a larger amount of food, the user may be given the illusion that they have eaten a larger amount of food than they actually did, thereby increasing their sense of satisfaction with the meal. This is effective for users who are trying to lose weight, for example. Furthermore, by superimposing an image showing a smaller amount of food, the user may be given the illusion that they have eaten less food than they actually did, thereby encouraging them to eat the necessary amount of food. This is effective for users who have a small appetite, for example.

[0042] (How to set the overlap area) A method for setting a superimposition area by the superimposition area setting unit 203 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of setting a superimposition area. In the example of Fig. 5, the superimposition area setting unit 203 sets a superimposition area AR1 in an image 51 captured by the image capturing unit 22, using a three-dimensional code OB1 as a reference.

[0043] The three-dimensional code OB1 is attached to a table on which the lunchbox is placed. The three-dimensional code OB1 indicates the position and extent of the chicken breast contained in the lunchbox. More specifically, the three-dimensional code OB1 contains information indicating the position of the three-dimensional code OB1 in three-dimensional space and the position and extent of the chicken breast (the area in which the chicken breast is captured). This information may be expressed, for example, in three-dimensional spatial coordinates.

[0044] The superimposition area setting unit 203 can identify the position and area in image 51 where the chicken breast appears, based on the information included in the three-dimensional code OB1, and set the superimposition area at that position and area. For example, the information included in the three-dimensional code OB1 may indicate the three-dimensional coordinate values ​​of the four corners of a rectangle indicating the position and area of ​​the chicken breast. In this case, the superimposition area setting unit 203 can identify the position and area in image 51 where the chicken breast appears by converting the coordinate values ​​into coordinate values ​​on image 51. The lunch box needs to be placed at a predetermined position relative to the three-dimensional code OB1. The three-dimensional code OB1 may also include information related to the food (e.g., the name of the dish, the ingredients used, etc.), information indicating an image to be superimposed on the food, etc.

[0045] It is also possible to set the superimposition area in a similar manner using other types of codes, such as two-dimensional codes. The item to which the code is attached may be any item that is in a predetermined positional relationship with the food. For example, the code may be attached to an item placed on a table, such as a tablecloth, or to a container or dish that holds food.

[0046] Here, for example, a restaurant or the like may offer a variety of foods, and it may be necessary to vary the superimposition area depending on the type of food. Therefore, for example, the superimposition area setting unit 203 may acquire order information indicating the ordered foods and set the superimposition area to a position and range corresponding to the ordered foods. In this case, codes may be attached to the table or items placed on the table, such as a tablecloth, and the ordered foods may be placed in predetermined positions according to the codes.

[0047] As described above, the detection target that serves as the basis for setting the superimposition area may be a code that indicates the position and range of the food, attached to an item that is in a predetermined positional relationship with the food. In this case, the superimposition area setting unit 203 sets the superimposition area based on the position and range indicated by the code. This provides the effect of being able to set the superimposition area with even greater precision, in addition to the effects achieved by the information processing device 1 according to the first exemplary embodiment.

[0048] The superimposition area does not necessarily have to be set in an area where food appears. For example, the superimposition area setting unit 203 may set the superimposition area outside the area where food appears when displaying information related to food, such as the calorie content of the food, or when displaying an image of a character. For example, the superimposition area setting unit 203 may set the superimposition area outside the area indicated by the code, along the outer edge of the position and area indicated by the code. This allows information related to the food to be displayed along the outer edge of the food. As described above, the character may be displayed as a still image or a moving image. Furthermore, the character may be a two-dimensional image or a three-dimensional image. For example, an image of a virtual character from an anime or manga may be superimposed, or an image of a real person (including a celebrity) may be superimposed.

[0049] (Regarding food category identification) Here, the overlapping area AR1 shown in Figure 5 includes areas where no chicken breast meat is visible. If an image of another food (for example, an image of fried chicken as shown in Figure 3) is overlapped in these areas, it will appear as if there is food in a location where there is no food actually present.

[0050] Therefore, when displaying an image of another food superimposed on a food, the superimposed image generation unit 205 may identify the food area in which the food appears within the superimposed area AR1, and generate a superimposed image according to the shape and size of the identified food area.

[0051] For example, in the example of Figure 5, the superimposition area AR1 is divided into 15 sections, 5 vertically and 3 horizontally. Of these sections, the section located at the bottom left corner contains almost no chicken breast. Therefore, the superimposition image generation unit 205 may identify the sections containing food as food areas, and may identify sections containing no food or where the proportion of the food area in the section is below a threshold as non-superimposition areas. This makes it possible to superimpose only the food area within the superimposition area, rather than the entire superimposition area.

[0052] Note that any method for identifying a food region may be used, and for example, if the color of the food is known, superimposed image generation unit 205 may identify the food region based on the color (specifically, pixel value). Alternatively, for example, superimposed image generation unit 205 may detect edges in image 51, which are portions where pixel values ​​change suddenly, and identify regions on the high pixel value side or low pixel value side of the edge as food regions.

[0053] When identifying a food region based on color, the color range of the food can be determined in advance. Note that the same food may appear in different colors depending on the lighting conditions, etc. For this reason, multiple color ranges may be set for one food. In this case, the superimposed image generation unit 205 sets the food region as an area consisting of pixels corresponding to each color range.

[0054] (Method of generating superimposed images and color correction) A method for generating a superimposed image by the superimposed image generating unit 205 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of generation of a superimposed image. Color correction by the correcting unit 204 will also be described here.

[0055] Image 61 shown in FIG. 6 is an image captured by the image capture unit 22, and shows a lidded lunchbox containing chicken breast. As shown in the figure, a three-dimensional code OB1 and a color correction marker M1 are attached to the inside of the lid of this lunchbox (the side exposed when the lid is opened). In this way, the detection target (in this example, the three-dimensional code OB1) that serves as a reference for setting the overlapping area may be attached to the container containing the food. In this case, it is desirable to attach the detection target in a position parallel to the surface of the table on which the lunchbox is placed.

[0056] (About color correction) As described above, identifying a food region enables superimposition display according to the shape of the food region, but the accuracy of identifying the food region may decrease due to the influence of shooting conditions such as the lighting environment. For example, when identifying a region consisting of pixel values ​​equal to or greater than a predetermined threshold as a food region, if the area around the food is dark, there is a risk that an area smaller than the area in which the food actually appears may be identified as the food region.

[0057] Therefore, the correction unit 204 performs color correction on the image 61 using the color correction marker M1, which is detected from the image 61 and serves as a reference for color correction. Then, the display control unit 206 superimposes and displays an image of a predetermined food in an area of ​​a predetermined color range within the superimposed area after color correction. This makes it possible to cancel the influence of the shooting conditions of the image 61, and therefore, in addition to the effect achieved by the information processing device 1 according to the first exemplary embodiment, it is possible to obtain the effect of being able to stably superimpose and display in an appropriate position and range regardless of the shooting conditions.

[0058] Color correction is performed so that the pixel values ​​of the color correction marker M1 become predetermined values. For example, the color correction marker M1 shown in FIG. 6 includes multiple color samples. More specifically, the left end of the color correction marker M1 is a white color sample, the right end is a black color sample, and multiple gray color samples (grays closer to black toward the right) are arranged between these color samples. Note that it is sufficient for the color correction marker M1 to include at least one color sample.

[0059] For example, suppose the pixel values ​​(RGB values) of the white color sample area in image 61 are (230, 216, 231). In this case, correction unit 204 corrects image 61 so that the pixel value of the pixel becomes a predetermined value (e.g., (239, 239, 239)). This makes it possible to cancel the influence of the shooting conditions of image 61 and identify the food area. The predetermined value may be determined in advance.

[0060] Note that instead of correcting the color of the image 61, the correction unit 204 may use a color correction marker M1 to correct the threshold used to identify the food region. For example, if the pixel value of the color sample of the color correction marker M1 shown in the image 61 is larger than a reference value, the correction unit 204 may also correct the threshold to a value larger than the reference value. The same applies when the pixel value of the color sample is smaller than the reference value. For example, when correcting the threshold for a color image 61, the correction unit 204 may correct the threshold for each of the R, G, and B values. The amount of threshold correction may be determined according to the difference between the pixel value of the color sample and the reference value. This type of correction can also cancel the influence of the shooting conditions of the image 61.

[0061] Alternatively, multiple correction patterns may be prepared in advance according to the shooting conditions. In this case, correction unit 204 may correct image 61 by applying a correction pattern according to the shooting conditions when image 61 was captured. Similarly, a threshold value according to the shooting conditions when image 61 was captured may be applied from among multiple threshold values ​​according to the shooting conditions to identify the food region. When these configurations are employed, color correction markers are not required.

[0062] In order to improve the accuracy of identifying food regions, it is preferable that the color of the container that holds the food and the dish on which the food is placed be easy to distinguish from the food. For example, if the food is close to white, the container and dish should be black.

[0063] (Method for generating superimposed images) Next, the method for generating the superimposed image will be explained. As described above, the superimposed area setting unit 203 sets the superimposed area AR1 based on the three-dimensional code OB1. After this, the superimposed image generation unit 205 cuts out the portion of the superimposed area AR1 from the image 61 to generate a partial image. The superimposed image generation unit 205 then identifies the food area in the generated partial image, processes an image of another food to match the shape and size of the identified food area, and combines the processed image with the food area. The superimposed image generation unit 205 also makes the background area of ​​the partial image (the area outside the food area) transparent, thereby completing the superimposed image. The background area may be made semi-transparent. By making it semi-transparent, the shading of ingredients and the like can be reflected in the processed superimposed image.

[0064] Here, in image 61, the lunchbox is not facing directly relative to the orientation of image 61. Therefore, the three-dimensional code OB1 appears tilted in image 61, and as a result, a tilted superimposition area AR1 is set. More specifically, the superimposition area AR1 in image 61 is tilted to the right by an angle θ with respect to the up-down direction of image 61.

[0065] In such a case, the display control unit 206 may display, in the superimposition area AR1, an image rotated by an angle corresponding to the angle of inclination of the superimposition area AR1. This provides the effect of being able to display a good-looking superimposed image even when the superimposition area AR1 is inclined, in addition to the effect provided by the information processing device 1 according to the first exemplary embodiment.

[0066] When displaying the rotated image in the superimposed area AR1, the superimposed image generation unit 205 cuts out the portion of the superimposed area AR1 from the image 61, and then rotates and transforms that portion (specifically, rotates it leftward by an angle θ) to generate the partial image 62. Next, the superimposed image generation unit 205 identifies the food area AR2 in the partial image 62. The food area AR2 can be identified by color or edge detection, as described above.

[0067] Next, the superimposed image generation unit 205 processes the image of the other food to fit the shape and size of the food area AR2, and then combines the processed image with the food area AR2. The superimposed image generation unit 205 also makes the background area AR3 of the food area AR2 in the partial image 62 transparent, thereby completing the superimposed image. Making it transparent means setting the transparency to 100%. The superimposed image can also be called a mask image.

[0068] Then, the display control unit 206 generates image 63 by projectively transforming the superimposed image generated as described above so that it fits into the superimposition area AR1 in image 61, and combines this image 63 with the superimposition area AR1 of image 61. As a result, an image rotated by an inclination angle corresponding to the inclination angle of the superimposition area AR1 is displayed in the superimposition area AR1.

[0069] (Flow of display control method) The flow of the display control method executed by the information processing device 2 will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing the flow of the display control method. In the following, an example will be described in which, when a food is placed in front of a user wearing the information processing device 2, an image of another food is superimposed on the image of the food.

[0070] In S21, the image acquisition unit 201 acquires an image captured by the imaging unit 22, that is, an image capturing at least a part of the visual field of the user wearing the information processing device 2. This image captures the food that the user intends to eat.

[0071] In S22, the detection unit 202 detects a predetermined detection target from the image acquired in S21. Then, in S23, the superimposition area setting unit 203 sets a superimposition area for information related to food (specifically, images of other foods) in the image acquired in S21, based on the detection target detected in S22.

[0072] For example, suppose that in S22 the detection unit 202 detects a three-dimensional code OB1 as shown in Fig. 6. In this case, in S23 the superimposition area setting unit 203 sets the superimposition area at the position and range indicated by the three-dimensional code OB1. As described with reference to Fig. 6, the superimposition area set in this manner may be tilted.

[0073] Therefore, in S24, the superimposed image generation unit 205 generates a partial image in which the tilt is canceled by cutting out the portion of the superimposed area set in S23 from the image acquired in S21 and performing rotational transformation on it. Note that if the superimposed area set in S23 is not tilted, the processing of S24 can be omitted.

[0074] In S25, the detection unit 202 detects color correction markers from the image acquired in S21. Then, in S26, the correction unit 204 performs color correction on the partial image generated in S24 using the color correction markers detected in S25. Note that color correction may be performed on the image acquired in S21. Furthermore, the processes of S25 and S26 can be performed at any timing before S27.

[0075] In S27, the superimposed image generation unit 205 identifies an area within a predetermined color range in the color-corrected partial image as a food area in which food appears. Then, in S28, the superimposed image generation unit 205 processes an image of another food to match the shape and size of the identified food area, and combines the processed image with the food area. The superimposed image generation unit 205 also makes the background area of ​​the partial image (the area outside the food area) transparent, thereby completing the superimposed image.

[0076] In S29, the display control unit 206 superimposes an image of another food on the superimposition area set in S23. Specifically, the display control unit 206 performs projection transformation on the superimposition image generated in S28 to fit the superimposition area in the image acquired in S21, and then combines the image with the superimposition image. This makes the food placed in front of the user wearing the information processing device 2 appear as another food to the user. The process in FIG. 7 is performed each time the image acquisition unit 201 acquires a new image, thereby realizing superimposition display in real time.

[0077] (Regarding the setting of multiple food categories) It is also possible to set multiple food regions within the superimposed region. This will be explained with reference to Figure 8. Figure 8 is a diagram showing an example in which multiple food regions are set within the superimposed region and superimposed on each other. Figure 8 shows an image captured by the image capturing unit 22, i.e., image 81 capturing at least a portion of the field of view of the user wearing the information processing device 2, and image 82 in which images of other foods are superimposed on each of the food regions set in image 81.

[0078] Image 81 shows food served on a plate. A three-dimensional code OB1 is attached to the plate, and a circular overlapping area AR1 is set inside the plate using this three-dimensional code OB1 as a reference. Multiple food areas AR2 are also set within overlapping area AR1.

[0079] In this example, the superimposed image generation unit 205 sets a rectangular area that includes an area made up of pixels in a predetermined color range within the superimposed area AR1 as the food area AR2. This allows the food area AR2 to be set to target some of the ingredients contained in the food. For example, by setting a rectangular area that includes an area made up of orange pixels as the food area AR2, the area in which an orange ingredient (e.g., a carrot) appears can be set as the food area AR2.

[0080] Then, the superimposed image generating unit 205 generates a superimposed image using images of other ingredients, and the display control unit 206 displays this superimposed image superimposed on the food area AR2, so that the user wearing the information processing device 2 recognizes the ingredients in the food area AR2 as other ingredients, as shown in image 82. This makes it possible to eat ingredients that the user dislikes, or to increase satisfaction by making it appear as if the food contains ingredients that the user likes.

[0081] (Example of detecting hands / tableware) The detection target detected by the detection unit 202 may be the hand of the user wearing the information processing device 2. Alternatively, the detection target detected by the detection unit 202 may be a dish held by the user. In this case, the superimposition area setting unit 203 may set the superimposition area based on the user's hand or the dish held by the user. This provides the effect of being able to display a superimposed image on food the user is about to eat, in addition to the effect provided by the information processing device 1 according to exemplary embodiment 1.

[0082] Fig. 9 is a diagram showing an example of a superimposed image displayed with chopsticks held by a user as the detection target. Fig. 9 shows an image 91 captured by the image capturing unit 22, i.e., an image of at least a part of the field of view of a user wearing the information processing device 2, and an image 92 in which an image of another food is superimposed in a superimposed area set in image 91.

[0083] In the example of FIG. 9, the detection unit 202 detects a pair of chopsticks OB2. When chopsticks are the detection target, it is preferable to have the user use chopsticks whose color is easily distinguishable from food. The overlapping area setting unit 203 then sets a rectangular overlapping area AR1 at the tip of the detected chopsticks OB2. Within the overlapping area AR1, chicken breast being held by the chopsticks OB2 is captured. The shape of the overlapping area AR1 is arbitrary and is not limited to a rectangle.

[0084] Here, the superimposed image generating unit 205 generates a superimposed image using images of other ingredients, and the display control unit 206 displays this superimposed image in the superimposed area AR1. As a result, as shown in image 92, the user wearing the information processing device 2 recognizes that the ingredient (chicken breast) shown in the superimposed area AR1 is the other ingredient (fried chicken).

[0085] FIG. 10 also shows an example of setting an overlapping area for a fork as a detection target, and an example of setting an overlapping area for a user's hand holding the fork as a detection target. In the fork shown as 101 in FIG. 10, the neck is marked in a different color from the other parts and designated as marker OB. The detection unit 202 detects this marker OB3 as a detection target. In this way, a part of the tableware can also be used as a detection target. Because marker OB3 is located at the tip of the fork handle, it can be detected even while the fork is being held. Furthermore, because it is located closer to the user than the tip of the fork, it can be detected even when food is stuck on the fork. Marker OB3 may be detected based on color alone, or a combination of color and shape.

[0086] The length of the marker OB3 on the fork is L1, and the length of the portion of the fork closer to the tip than the marker OB3 is L2. Because the ratio of these lengths remains the same in the image of the fork, the superimposition area setting unit 203 can identify the position where the tip of the fork appears based on the orientation and length of the marker OB3 detected in the image and the ratio, and can set the superimposition area AR1 as shown in Figure 10 using that position as a reference.

[0087] A code such as a three-dimensional code may be attached to the fork as a detection target. In this case, the code may include information indicating the position of the tip of the fork or information indicating the position and range of the overlapping area. As in the example of Figure 9, the entire fork may be the detection target, and the overlapping area AR1 may be set at the tip. The shape of the overlapping area AR1 may be any shape and is not limited to a circle. The same applies to 102 in Figure 10, which will be described below.

[0088] 102 in FIG. 10 shows an example of setting the superimposition area AR1 with the user's hand OB4 holding a fork as the detection target. Because the position of the hand when bringing food to the mouth and the position of the food have a predetermined positional relationship, it is possible to set the superimposition area AR1 on the food by setting hand OB4 as the detection target. For example, as shown in 102 in FIG. 10, the superimposition area setting unit 203 may set the superimposition area AR1 at a position inside hand OB4 (to the left of hand OB4 if it is a right hand, or to the right of hand OB4 if it is a left hand). The distance from hand OB4 to set the superimposition area AR1 may be determined depending on the type of tableware and food being used, etc.

[0089] [Modification] The execution entity of each process described in each of the above exemplary embodiments is arbitrary and is not limited to the above examples. In other words, a display control system having the same functions as the information processing devices 1 and 2 can be constructed using multiple devices that can communicate with each other. For example, in the block diagrams of Figures 1 and 4, a display control system having the same functions as the information processing devices 1 and 2 can be constructed by distributing each block shown in those block diagrams across multiple devices.

[0090] 4, blocks from the image acquisition unit 201 to the superimposed image generation unit 205 included in the control unit 20 may be provided in a server. In this case, the information processing device 2 transmits an image captured by the image capture unit 22 to the server, and receives and displays the image with information superimposed on it from the server. With such a display control system, it is also possible to have one server control the displays of multiple information processing devices 2.

[0091] [Software implementation example] Some or all of the functions of the information processing devices 1 and 2 may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0092] In the latter case, the information processing devices 1 and 2 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 11. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program (display control program) P for operating the computer C as the information processing device 1 or 2. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing device 1 or 2.

[0093] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0094] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0095] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0096] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0097] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0098] (Appendix 1) An information processing device comprising: a detection means for detecting a predetermined detection target from an image capturing at least a portion of a user's field of view, the image including food; an overlay area setting means for setting an overlay area for information related to the food in the image based on the detection target; and a display control means for superimposing the information on the overlay area.

[0099] (Appendix 2) The information processing device according to claim 1, wherein the display control means causes an image of a predetermined food to be superimposed on the superimposition area.

[0100] (Appendix 3) a correction unit that performs color correction on the image using a color correction marker that is detected from the image and serves as a reference for color correction; The information processing device according to claim 2, wherein the display control means superimposes and displays the image of the predetermined food in an area of ​​a predetermined color range within the superimposed area after color correction.

[0101] (Appendix 4) An information processing device described in any one of Appendices 1 to 3, wherein the detection object is the user's hand or tableware held by the user, and the overlapping area setting means sets the overlapping area based on the user's hand or tableware held by the user.

[0102] (Appendix 5) An information processing device described in any one of Appendices 1 to 4, wherein the detection object is a code indicating the position and range of the food, attached to an item that is in a predetermined positional relationship with the food, and the overlapping area setting means sets the overlapping area based on the position and range indicated by the code.

[0103] (Appendix 6) The information processing device according to claim 5, wherein the display control means displays an image rotated by an inclination angle corresponding to the inclination angle of the superimposition area in the superimposition area.

[0104] (Appendix 7) A display control method including: at least one processor detecting a predetermined detection object from an image capturing at least a portion of a user's field of view, the image including food; setting a superimposition area for information related to the food in the image based on the detection object; and superimposing the information in the superimposition area.

[0105] (Appendix 8) A display control program that causes a computer to function as a detection means for detecting a predetermined detection target from an image that captures at least a portion of a user's field of vision and that includes food, an overlay area setting means for setting an overlay area for information related to the food in the image based on the detection target, and a display control means for superimposing and displaying the information in the overlay area.

[0106] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0107] An information processing device comprising at least one processor, the processor executing a detection process for detecting a predetermined detection target from an image capturing at least a portion of a user's field of view, the image including food; a superimposition area setting process for setting a superimposition area for information related to the food in the image based on the detection target; and a display control process for superimposing the information on the superimposition area.

[0108] The information processing device may further include a memory that stores a program for causing the processor to execute the detection process, the superposition area setting process, and the display control process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0109] 1, 2 Information processing device 11, 202 Detector 12, 203 Superimposition area setting section 204 Correction Unit 13, 206 Display control unit OB1 3D code (code, detection target) OB2 Chopsticks (detection target) OB3 marker (detection target) OB4 Hand (detection target) M1 Color Correction Marker

Claims

1. a detection means for detecting a predetermined detection target from an image capturing at least a part of a user's visual field, the image including food; a superimposition area setting means for setting a superimposition area of ​​a superimposed image including information related to the food in the image based on the detection target; a superimposed image generating means for generating the superimposed image; a display control means for displaying the superimposed image in the superimposed area in a superimposed manner, the display control means causes an image of a predetermined food to be superimposed on the superimposed area; The superimposed image generating means Identifying a food area and a background area from the image in which the food appears; processing the image of the predetermined food to fit the shape and size of the identified food area; Composite the processed image of the predetermined food with the food area; generating the superimposed image by making the background region transparent or semi-transparent; Information processing device.

2. the food region comprises a plurality of food regions; The superimposed image generating means processing the image of the predetermined food to fit the shape and size of each of the identified food regions; generating the superimposed image by superimposing the processed image of the predetermined food onto the portions of the plurality of food regions; The information processing device according to claim 1 .

3. a correction unit that performs color correction on the image using a color correction marker that is detected from the image and serves as a reference for color correction; the display control means causes the image of the predetermined food to be superimposed on an area of ​​a predetermined color range within the superimposed area after color correction; The information processing device according to claim 2 .

4. the detection target is the user's hand or a dish held by the user, the overlapping area setting means sets the overlapping area based on the user's hand or the tableware held by the user; The information processing device according to claim 1 .

5. the detection target is a code that indicates the position and range of the food, and that is attached to an item that is in a predetermined positional relationship with the food; the superimposition area setting means sets the superimposition area based on the position and range indicated by the code. The information processing device according to claim 1 .

6. the display control means displays, in the superimposed area, an image rotated by an inclination angle corresponding to the inclination angle of the superimposed area; The information processing device according to claim 5 .

7. At least one processor Detecting a predetermined detection target from an image capturing at least a part of a user's field of view, the image including food; setting a superimposed area of ​​a superimposed image including information related to the food in the image based on the detection target; generating the superimposed image; and displaying the superimposed image in the superimposed area in a superimposed manner, The superimposed display includes superimposing an image of a predetermined food on the superimposed area, The superimposed image is Identifying a food area and a background area from the image in which the food appears; processing the image of the predetermined food to fit the shape and size of the identified food area; Composite the processed image of the predetermined food with the food area; Generated by making the background region transparent or semi-transparent. Display control method.

8. Computer, a detection means for detecting a predetermined detection target from an image capturing at least a part of the user's field of view, the image including food; a superimposition area setting means for setting a superimposition area of ​​a superimposed image including information related to the food in the image based on the detection target; a superimposed image generating means for generating the superimposed image; and functioning as a display control means for displaying the superimposed image in the superimposed area; the display control means causes an image of a predetermined food to be superimposed on the superimposed area; The superimposed image generating means Identifying a food area and a background area from the image in which the food appears; processing the image of the predetermined food to fit the shape and size of the identified food area; Composite the processed image of the predetermined food with the food area; generating the superimposed image by making the background region transparent or semi-transparent; Display control program.

Images