Display control method

Abstract

To provide a display control method that enables more appropriate calculation and notification of a physical quantity of an object.SOLUTION: A display control method for a display device (100) includes: detecting a type of a real object included in a captured image generated by a camera (111) of the display device; measuring, for each of a plurality of measurement points on the real object, a distance to each measurement point; recognizing a three-dimensional shape of the real object on the basis of the distance to estimate a volume of the real object; receiving an input of a required mass of the real object; calculating a required volume corresponding to the required mass on the basis of a density corresponding to the type of the real object; and displaying, on a display (102) of the display device, a guide marker image that indicates a region of the three-dimensional shape of the real object so as to be superimposed on the real object by calculating the region corresponding to the calculated required volume.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display control method. [Background technology]

[0002] Patent Document 1 discloses an example of an information processing device with a weighing assist function, which is an information processing device (abstract excerpt) that includes: "an estimation unit that estimates the amount of at least one of the ingredients to be cooked and the seasonings to be used in cooking based on a signal detected by a sensor; an index calculation unit that calculates a predetermined cooking index according to the estimation result by the estimation unit; and a notification control unit that controls the notification of the cooking index calculated by the index calculation unit." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2014 / 132521 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, object identification and mass estimation are performed based on information obtained from imaging and odor sensors and various concentration sensors (salt sensors and sugar sensors) installed on external cooking appliances, so if, for example, multiple different objects are the same color and no differences are recognized from the information detected by the sensors, it is not possible to distinguish between the objects. In addition, it is necessary to prepare an external device equipped with a sensor separately.

[0005] In addition, since the information obtained by imaging is two-dimensional, the size of the object cannot be accurately determined, and there is a risk that the estimated mass will deviate significantly from the actual value.

[0006] Therefore, an object of the present invention is to provide a display control method that can more suitably calculate and notify the physical quantity of an object. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention has the configurations described in the claims. As an example, the present invention is a display control method for a display device, comprising the steps of: detecting a type of a physical object included in a captured image generated by a camera of the display device; measuring distances to each of a plurality of measurement points on the physical object; recognizing a three-dimensional shape of the physical object based on the distances to each of the measurement points; estimating a volume of the physical object based on the three-dimensional shape of the physical object; accepting an input of a required mass that is a mass of the physical object; calculating a required volume corresponding to the accepted required mass based on a density corresponding to the detected type of the physical object; calculating a region of the three-dimensional shape of the physical object corresponding to the calculated required volume based on the estimated volume of the physical object; and displaying a reference marker image indicating the calculated region on a display of the display device so as to be superimposed on the physical object. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a display control method that can more suitably calculate and notify the physical quantity of an object. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overview of an augmented reality display device according to a first embodiment. [Figure 2] FIG. 1 is a hardware configuration diagram of an HMD. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of a weighing assistance program executed by a processor of the HMD. [Figure 4] 4 is a flowchart showing the operation of the HMD according to the first embodiment. [Figure 5] 4 is a flowchart showing the operation of the HMD according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a rescan notification display. [Figure 7] FIG. 10 is a diagram showing an overview of an augmented reality display device according to a second embodiment. [Figure 8] 10 is a flowchart showing the operation of an HMD according to the second embodiment. [Figure 9] 10A and 10B are diagrams showing examples of switching the display position of a guide marker. [Figure 10] FIG. 10 is a diagram showing an overview of an augmented reality display device according to a third embodiment. [Figure 11A] FIG. 10 is a diagram showing an overview of an augmented reality display device according to a third embodiment. [Figure 11B] FIG. 11 is a diagram showing an example of a screen display used in a modification of the third embodiment. [Figure 11C] 10 is a flowchart showing the operation of a modified example of the third embodiment. [Figure 12A] FIG. 1 is a diagram showing an example of the configuration of an augmented reality display system. [Figure 12B] FIG. 1 is a diagram showing an example of the configuration of an augmented reality display system. [Figure 13A] FIG. 10 is a diagram showing an example of information output from an HMD. [Figure 13B] FIG. 10 is a diagram showing an example of information output from an HMD. [Figure 14] FIG. 1 illustrates an example of an augmented reality display system. [Figure 15A] FIG. 10 is a diagram showing an example of an AR image display when there are multiple target objects. [Figure 15B] FIG. 10 is a diagram showing an example of an AR image display when there are multiple target objects. [Figure 16] FIG. 10 is a diagram showing an example of AR display in a situation where multiple target objects are visible overlapping each other. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention is expected to contribute to diversification and technological improvement in labor-intensive industries, for example, and therefore contributes to the achievement of Sustainable Development Goal 8.2 (increasing economic productivity through diversification, technological improvement, and innovation, particularly in labor-intensive industries and industries that increase the value of goods and services) advocated by the United Nations. Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0011] First Embodiment The first embodiment is an embodiment in which a head-mounted display is used as an augmented reality display device, and a measurement assistance function for a real object is used to display the physical quantity (at least one of volume and mass) of the real object on the display as an augmented reality image.

[0012] FIG. 1 is a diagram showing an overview of the augmented reality display device according to the first embodiment.

[0013] 1 is equipped with a see-through display 102. A user 1 wearing the HMD 100 visually recognizes a real object viewed through the display 102, with an augmented reality image (hereinafter referred to as an "AR image") displayed on the display 102 superimposed on the real object.

[0014] When the HMD 100 recognizes target objects 200, 201, 202, 203, and 204 for which physical quantities are to be estimated among the real objects captured by the outer camera 111 mounted on the HMD 100, the HMD 100 displays frames 210, 211, 212, 213, and 214 surrounding the target objects.

[0015] The HMD 100 then estimates the physical quantities of each of the target objects 200, 201, 202, 203, and 204, and displays the estimated results in AR images in measurement result fields 220, 221, 222, 223, and 224 on the display 102. At this time, the measurement result fields 220, 221, 222, 223, and 224 are displayed on the display 102 so as to be displayed near the target objects 200, 201, 202, 203, and 204.

[0016] Each measurement result field 220, 221, 222, 223, and 224 displays the name of the target object and the physical quantity of the target object. The physical quantity here may be the volume of the target object, or, if the density of the target object is known, may be the mass converted from the density and volume. In this embodiment, mass and volume are used as the physical quantities.

[0017] In addition to the target objects 200, 201, 202, 203, and 204, the image captured by the outer camera 111 also includes a physical object 205. However, since the HMD 100 does not recognize the physical object 205 as a target object, no frame is added to the physical object 205, and no estimation of its physical quantities and display of the results are performed.

[0018] If the user 1 wants to correct the results of the recognition of the types of the target objects 200, 201, 202, 203, and 204 by the HMD 100 from the image captured by the outer camera 111, the user 1 may input voice information from the microphone 121 of the HMD 100, or an AR image such as that shown in the character input UI 131 may be displayed on the display 102, and the outer camera 111 or the distance sensor 167 may recognize the user 1's gesture movements to input characters, thereby inputting the type correction information.

[0019] FIG. 2 is a diagram showing the hardware configuration of the HMD 100.

[0020] The HMD 100 is composed of a processor 101, a display 102, a ROM 103, a RAM 104, a storage 105, an outer camera 111, an inner camera 112, a microphone 121, a speaker 122, an operation button 130, a wireless LAN communicator 141, a close-proximity wireless communication device 142, a telephone network communicator 143, an expansion I / F 150, a group of sensors 160, and a battery 180, all of which are interconnected via a bus 106.

[0021] The sensor group 160 may include a GPS (Global Positioning System) 161, a gyro sensor 162, a geomagnetic sensor 163, an acceleration sensor 164, an illuminance sensor 165, a proximity sensor 166, a distance measurement sensor 167, and a line of sight detection sensor 168.

[0022] The distance measurement sensor 167 may be a ToF (Time Of Flight) sensor or an ultrasonic sensor, or if the outer camera 111 is a stereo camera, the outer camera 111 may be used as the distance measurement sensor 167 because the distance to a real object can be measured using left-right parallax. Even if the outer camera 111 is a ToF camera, the outer camera 111 can be used as the distance measurement sensor 167.

[0023] The ROM 103 or the storage 105 stores a weighing assistance program for the HDM 100.

[0024] FIG. 3 is a block diagram showing an example of the configuration of a weighing assistance program executed by the processor 101 of the HMD 100. As shown in FIG.

[0025] The measurement assist program includes a range image data acquisition unit 11, a range image data storage unit 12, an object detection unit 13, a target object identification unit 14, a type estimation unit 15, a type dictionary storage unit 16, a physical quantity estimation unit 17, a display control unit 18, a communication control unit 19, and a density dictionary storage unit 20. The physical quantity estimation unit 17 includes a density acquisition unit 17a, a 3D shape acquisition unit 17b, and a mass calculation unit 17c. The processor 101 loads the measurement assist program into RAM 104 and executes it, thereby realizing the functions of each of the above units. The processor 101 may also be configured as an integrated circuit that realizes the same functions as the measurement assist program. Details of the processing of each of the above units will be described with reference to the flowcharts in Figure 4 and subsequent figures.

[0026] Alternatively, the HMD 100 and a server or information processing device may execute a weighing assistance program, and the results may be returned to the HMD 100 and displayed on the display 102.

[0027] The operation of the HMD 100 according to the first embodiment will be described with reference to Fig. 4 to Fig. 6. Fig. 4 and Fig. 5 are flowcharts showing the operation of the HMD 100 according to the first embodiment. Fig. 6 is a diagram showing an example of a rescan notification display.

[0028] 4 starts when the main power of the HMD 100 is turned on. The outer camera 111 captures an image of the outside world and outputs the captured image to the distance image data acquisition unit 11 (S101).

[0029] Distance measurement sensor 167 has a distance measurement area that includes the angle of view of outer camera 111. Distance measurement sensor 167 performs distance measurement in synchronization with outer camera 111 and outputs distance measurement data to distance image data acquisition unit 11 (S101). Distance image data acquisition unit 11 stores distance image data that associates imaging data with distance data in distance image data storage unit 12 (S102).

[0030] The object detection unit 13 reads out the distance image data from the distance image data storage unit 12. The object detection unit 13 performs a subject detection process on the distance image data to detect a real object (subject) captured in the captured image (S103).

[0031] When the object detection unit 13 detects at least one subject (S103: Yes), and the user 1 selects a real object (referred to as a "target object") for measuring a physical quantity (S104: Yes), the type estimation unit 15 estimates the type of the target object (S105).

[0032] As an example of a method for selecting a target object, a mode using gaze detection sensor 168 will be described. Gaze detection sensor 168 acquires a facial image of user 1 captured by in-camera 112 and detects the gaze from the area where the eyes are captured. If the gaze remains on a physical object detected by object detection unit 13 for a certain period of time or more, target object identification unit 14 recognizes that the physical object has been selected as the target object.

[0033] As another example of the selection method, the object detection unit 13 displays on the display 102 a plurality of frames 210, 211, 212, 213, and 214, each containing an area in which a real object detected from the imaging data is captured. When the user 1 makes a gesture to designate the frames 210, 211, 212, 213, and 214, the gesture is captured in the image captured by the outer camera 111. The target object identification unit 14 analyzes the gesture based on the range image data and recognizes whether or not a target object has been selected. When input is made by a gesture, the outer camera 111 corresponds to an example of an information input device.

[0034] Alternatively, the target object identification unit 14 may recognize the sound data collected by the microphone 121 and determine whether the target object has been selected. Therefore, the microphone 121 corresponds to an example of an information input device.

[0035] If the object detection unit 13 does not detect any subject (S103: No), or the target object identification unit 14 does not accept the selection of a target object (S104: No), and if the weighing assistance function of the HMD 100 is to be used continuously (S114: No, see Figure 5), new captured images and distance measurement data are acquired (S101, S102).

[0036] The type estimation unit 15 extracts image features such as the shape and color of the target object and estimates the type of the target object by comparing them with the image features of various real objects registered in the type dictionary storage unit 16. The type dictionary may be stored on a server that is communicatively connected to the HMD 100, and the HMD 100 may transmit a type determination request to the server as needed and receive the result.

[0037] Alternatively, the type estimation unit 15 may recognize characters, figures, or symbols written on the surface of the target object or on a package placed around the target object, and estimate the type of the target object.

[0038] The type estimation unit 15 notifies the user 1 of the estimation result of the type of the target object by displaying it on the display 102 or by outputting a sound from the speaker 122. If the user 1 approves the estimation result (S106: Yes), the type of the target object is confirmed (S108).

[0039] If user 1 does not approve the estimation result (S106: No), user 1 inputs type correction information (S107). For example, if the type estimated by type estimation unit 15 is "sugar," user 1 may want to correct it to "granulated sugar" or "salt." The input of correction information may be received by collecting user 1's speech with microphone 121 and having type estimation unit 15 perform voice analysis processing, or by displaying an AR image such as that shown in character input UI 131 on display 102, recognizing user 1's gestures with out-camera 111 or distance measurement sensor 167, and having type estimation unit 15 analyze and process the input character input information. When the input of correction / supplemental information is received, the type of the target object is confirmed (S108).

[0040] The above-described processing from step S101 to step S108 is the target object type determination processing (shown as processing "A" in FIG. 4).

[0041] When the type of the target object is determined (S108), the density acquisition unit 17a of the physical quantity estimation unit 17 acquires the density of the type (S109). The density may be determined by referring to a density dictionary stored in a server to which the HMD 100 is connected for communication, or a density dictionary storage unit 20 may be provided in advance in the storage 105 of the HMD 100.

[0042] The type dictionary and density dictionary may be prepared according to the scene in which the HMD 100 is used. For example, when the HMD 100 is used as a cooking assistant, it is expected that it will be necessary to measure flour, breadcrumbs, and water in addition to various seasonings (e.g., soy sauce, sauce, butter, margarine, vegetable oil, sugar, salt, Chinese seasonings, etc.), so a dictionary storing the image features (color, shape, etc.) and densities of these ingredients may be prepared as a cooking dictionary. It is also preferable that the image features and densities of various vegetables, meats, and processed foods (tofu, etc.) as cooking ingredients be stored.

[0043] In addition, in a scene where the HMD 100 is used to assist in gardening work, it is preferable that the image features and densities of types of soil, such as black soil, Akadama soil, Kanuma soil, and leaf mold, be stored in the dictionary.

[0044] Additionally, in plastering work, for example, it is preferable that the dictionary stores image features and densities of real objects used in various scenes that require measurement, such as various plaster wall materials, water, cement, etc.

[0045] If the accuracy of the physical quantity estimation of HMD100 is sufficiently high, it may be used, for example, in pharmacies to measure various bases (e.g., Vaseline, Macrogol, Plastibase, etc.) and the drugs to be mixed with them, or for compounding drugs in research facilities.

[0046] Next, the three-dimensional shape acquisition unit 17b of the physical quantity estimation unit 17 reads the range image data from the range image data storage unit 12 and acquires the three-dimensional shape of the target object (S110). The distance measurement sensor 167 may measure the distance from the range measurement sensor 167 (HMD 100) to each of multiple measurement points on the target object, interpolate connecting lines connecting adjacent measurement points, and reconstruct a plane including the connecting lines to acquire the three-dimensional shape. Alternatively, the three-dimensional shape of the target object may be acquired by performing surface rendering from the coordinates of the measurement points. Acquisition of the three-dimensional shape does not require acquisition of the complete three-dimensional shape of the target object, and the acquisition level may be adjusted depending on the intended use of the HMD 100.

[0047] The three-dimensional shape acquisition unit 17b may determine whether or not it can acquire a three-dimensional shape by, for example, determining that it is impossible if imaging and ranging data of the target object is obtained from only one direction (the depth distance of the measurement point is within a judgment threshold for determining whether depth information is missing, for example, 1 cm), but determining that it is possible to acquire a three-dimensional shape if imaging and ranging data is available from at least two directions, preferably three or more directions, i.e., the front direction, back direction, and depth direction of the target object.

[0048] If the three-dimensional shape acquisition unit 17b can reconstruct the approximate three-dimensional shape of the target object (S110: Yes), the volume / mass calculation unit 17c calculates the volume of the three-dimensional shape (S111) and multiplies it by the density to calculate the mass (S112).

[0049] The volume / mass calculation unit 17c outputs the type and physical quantity of the target object to the display control unit 18, and the display control unit 18 generates an AR image in which the type and physical quantity of the target object are displayed in the measurement result field, and displays it on the display 102 (S113). If the measurement assistance function by the HMD 100 is to be continued (S114: No), return to step S101.

[0050] On the other hand, if the three-dimensional shape acquisition unit 17b determines that the approximate three-dimensional shape of the target object cannot be acquired (S110: No), it sends a rescan notification 250 to the user 1 to prompt acquisition of range image data of the target object from a different angle as shown in FIG. 6 (S115), and returns to step S101.

[0051] According to this embodiment, the volume of an object to be measured is calculated by three-dimensionally measuring the shape of the object using the output from the outer camera 111 and distance measurement sensor 167 mounted on the HMD 100. Furthermore, if the density is known, the mass of the object is calculated using the volume and density. This allows the HMD 100 alone to accurately calculate the physical quantity of the object (volume and mass converted based on the volume) and notify the user 1.

[0052] Second Embodiment The second embodiment is an embodiment in which, in addition to the first embodiment, a guide marker is displayed so as to appear as if it has been drawn.

[0053] Fig. 7 is a diagram showing an overview of an augmented reality display device according to a second embodiment. For each of the target objects 200 and 202 shown in Fig. 7, multiple reference markers 230 and 232 indicating the required amount specified by the user 1 for the target objects 200 and 202, rather than the total amount of the target objects 200 and 202, are displayed in AR, and the object type and the mass and volume corresponding to the specified required amount are displayed in required amount fields 220a and 222a. Frames 210 and 212 are added to the target objects 200 and 202.

[0054] Furthermore, frames 216 and 217 are displayed in AR for target objects 206 and 207, respectively. Multiple required quantities are specified for each of target objects 206 and 207. Therefore, target object 206 is displayed with reference markers 236a and 236b, and the physical quantities corresponding to each reference marker are displayed in measurement result fields 226a and 226b. Target object 207 is also displayed with reference markers 237a and 237b, and the physical quantities corresponding to each reference marker are displayed in measurement result fields 227a and 227b.

[0055] 8 is a flowchart showing the operation of the HMD 100 according to the second embodiment. In the second embodiment, steps S120 to S124 are added to the flowchart of the first embodiment.

[0056] When the density acquisition unit 17a acquires the density of the target object (S109), the three-dimensional shape acquisition unit 17b acquires the three-dimensional shape of the target object (S110: Yes). If the user 1 specifies the required amount of the target object (S120: Yes), the user 1 inputs voice from the microphone 121, or an AR image such as that shown in the character input UI 131 is displayed on the display 102, and the out-camera 111 or the distance measurement sensor 167 recognizes the gesture of the user 1 to input characters, thereby inputting the required amount of the target object (S121).

[0057] The volume / mass calculation unit 17c calculates the required volume of the target object from the density, the three-dimensional shape of the target object, and the required amount (S122).

[0058] The volume / mass calculation unit 17c calculates the position of a reference marker to be displayed on the target object from the three-dimensional shape and required volume of the target object (S123).

[0059] The volume / mass calculation unit 17c outputs the position of the reference marker, the type of the target object, and the mass and volume corresponding to the required amount to the display control unit 18, and displays the reference marker superimposed on the real object in AR, and also displays the measurement result fields 226a, 226b, 227a, and 227b near the real object in AR (S124). Then, the process proceeds to step S114.

[0060] On the other hand, if the user 1 does not specify the required amount of the target object (S120: No), the process proceeds from step S111 to step S114, as in the first embodiment.

[0061] According to this embodiment, when it is desired to measure a required amount of a target object that is less than the entire amount, the target object can be divided into the required amount by dividing the target object along the guide marker displayed on the target object in AR.

[0062] 9 is a diagram showing an example of switching the display position of the guide marker. The display of the guide marker may change the target area. For example, guide marker 236b is displayed in the left area of ​​target object 206 (for example, an apple piece) in the figure. However, if a guide marker display switching operation is performed after step S124, volume / mass calculation unit 17c executes step S123 again, and display control unit 18 may be configured to display a new guide marker 236c in an area different from the superimposed display area of ​​the initially displayed guide marker.

[0063] Similarly, in the initial display, the internal space of the target object 208 (e.g., a planter) is divided vertically and a guide marker 238a is displayed, but when the display of the guide marker is switched, a new guide marker 238b is displayed by dividing the internal space along the depth direction.

[0064] In addition, in the initial display, a guide marker 239a is displayed in which the target object 209 (e.g., a glass) is held horizontally in an upright position, but when an instruction to switch the display position of the guide marker is input, a guide marker 239b in which the surface of the liquid is held horizontally in a tilted position of the target object 209 may be displayed.

[0065] The type of guide marker to be displayed may be specified by user 1 through voice input from microphone 121. Alternatively, when type estimation unit 15 identifies a target object, volume / mass calculation unit 17c may determine the attributes of the target object and automatically select a guide marker corresponding to the attributes. For example, if the attribute of the target object is determined to be "solid: solid," a "guide marker along the outer surface of the target object" may be automatically selected; if the attribute is determined to be "solid: container," a "guide marker along the storage space of the container" may be automatically selected; and if the attribute is determined to be "liquid," a "guide marker with a horizontal water surface and a contour along the inner surface of the container" may be automatically selected. Furthermore, the display of the guide marker may be switched by displaying an AR image such as those shown in guide marker display switching UIs 132a, 132b, and 132c on display 102 and recognizing a gesture by user 1 using out-camera 111 or distance measurement sensor 167.

[0066] According to this embodiment, by specifying the required amount for the target object and using the reference marker, it is possible to extract only the required amount.

[0067] Third Embodiment In addition to the second embodiment, the third embodiment is an embodiment in which, when there is a container and a material (an example of a target object) to be placed in the container, shape information of the material and shape information of the container (an example of a target object) are detected, and the volume that will give the mass of the material an approximate specific value and the corresponding volume inside the container (space) are calculated, and a guide marker is attached to the container so that this can be determined.

[0068] FIG. 10 is a diagram illustrating an overview of an augmented reality display device according to a third embodiment. In FIG. 10, for example, assume that there is Akadama soil 300 as a material and a planter 330 as a container. A frame 310 indicating that the Akadama soil 300 has been recognized as a material is displayed in AR around the Akadama soil 300. The required amount of Akadama soil 300 entered by user 1 is displayed in a required amount field 320, displaying the type "Akadama soil," the required amount converted to mass, and the required amount converted to volume. A guide marker 340 indicating the depth of the planter 330 when the required amount of Akadama soil 300 is poured in is superimposed and displayed in AR.

[0069] The material may be a liquid as well as a solid. For example, the material may be liquid fertilizer 301 and the container may be a spoon 331. A frame 311 indicating that the liquid fertilizer 301 has been recognized as a material is displayed in AR on the liquid fertilizer 301. A required amount field 321 for the liquid fertilizer 301 displays the type of material, the required amount converted into mass, and the required amount converted into volume. A guide marker 341 indicating how deep the spoon 331 will be if the required amount of liquid fertilizer 301 is poured is displayed in AR superimposed on the spoon 331.

[0070] The required amount field 322 shows a state in which two required amounts of liquid fertilizer 301 have been input. Guide markers 342a and 342b corresponding to the two required amounts are displayed on a cup 332 serving as a container. The display of the guide markers may be switched by displaying an AR image such as that shown in the character input UI 131 on the display 102 and recognizing a gesture by the user 1 using the outer camera 111 or the distance sensor 167. When storing the weighing results, a material and container designation UI 133 and a material information recording necessity selection UI 134 may be displayed as AR images to accept input operations.

[0071] 11A is a flowchart showing the operation of the HMD 100 according to the third embodiment. In the third embodiment, steps S130 to S141 are added to the flowchart of the second embodiment.

[0072] When the user 1 performs voice input from the microphone 121, or when an AR image such as that shown in the material and container designation UI 133 is displayed on the display 102 and the user 1 performs a gesture to input designation of a material and a container (S130: Yes), the object detection unit 13 frames all detected objects. When the user 1 selects a frame 310 to select as a material using a gesture (S131), the type estimation unit 15 estimates the type of material, and the density acquisition unit 17a acquires the density of the material (S132). When the user 1 inputs the type of material using voice input, the type estimation unit 15 may estimate the type based on the input information. Frames attached to real objects that were not selected are hidden.

[0073] The three-dimensional shape acquisition unit 17b acquires the three-dimensional shape of the material (S133: Yes), and when the user 1 inputs the required amount of material (S134), the volume / mass calculation unit 17c calculates the required volume of the target object from the density of the material and the three-dimensional shape and required amount of the target object (S135).

[0074] Next, user 1 selects a container (S136). The container selection operation may involve user 1 pointing to the container using a gesture, which is recognized by object detection unit 13, or user 1 may select the frame of the material and the frame of the container in advance in step S131; any type of selection operation may be used.

[0075] When the three-dimensional shape acquisition unit 17b acquires the three-dimensional shape of the container (S137: Yes), the volume / mass calculation unit 17c calculates the position of a reference marker to be displayed on the container from the required volume of the material and the shape of the container (S138).

[0076] The volume / mass calculation unit 17c outputs the position of the guide marker, the type of material, and the mass and volume corresponding to the required amount to the display control unit 18, and displays the guide marker superimposed on the container in AR, and also displays required amount fields 320, 321 in AR near the container (S139).

[0077] Processor 101 displays material and container designation UI 133 and material information recording necessity selection UI 134. If recording is necessary (S140: Yes), the type, required amount, required amount ratio, and remaining amount of one or more materials are recorded (S141).

[0078] After recording the remaining amount (S141), or if recording is not necessary (S140: No), if continuing (S114: No), the image capturing (S101) and distance measurement (S102) are repeated. If ending (S114: Yes), the above series of processes are ended.

[0079] If the three-dimensional shape of the material cannot be acquired (S133: No), or if the three-dimensional shape of the container cannot be acquired (S137: No), a new imaging direction is notified (S115), and the process returns to steps S101 and S102.

[0080] Furthermore, if the user 1 does not specify a material or a container (S130: No), similarly to the second embodiment, the density of the target object is acquired (S109), and the process proceeds to process B, which leads to displaying the object type, mass, volume, and reference marker (S124). If the process is to be continued (S114: No), the image capturing (S101) and distance measurement (S102) are repeated. If the process is to be ended (S114: Yes), the series of processes described above is ended.

[0081] According to this embodiment, when there are materials and a container for storing the materials as target objects, it is possible to display a guide marker when the required amount of material has been stored in the container.

[0082] An example of processing in which, after information on the target object (material) is recorded in step S141, the recorded information is used will be described with reference to FIGS. 11B and 11C.

[0083] FIG. 11B is a diagram showing an example of a screen display used in a modification of the third embodiment.

[0084] 11B is a UI for confirming whether or not to call up a recording of a previously used material. When information requesting a call is entered in the material recording information call UI 135a, a material recording information UI 135b is displayed.

[0085] Furthermore, if the weighing result indicates that there is a shortage of material, a material shortage notification UI 136a is displayed, and further, a material shortage amount display UI 136b is displayed, which indicates the shortage amount of the missing material. Next, a material addition necessity input UI 137a is displayed for confirming whether or not additional material is required, and if "Yes" is selected (S155: Yes), an additional material amount display UI 137b is displayed, and the user 1 selects the material to be added and inputs the amount to be added (S156). The additional amount may be input by the user 1 using voice input from the microphone 121 of the HMD 100, or by displaying an AR image such as that shown in the character input UI 131 on the display 102 and inputting characters by recognizing a gesture movement of the user 1 using the out-camera 111 or the distance measurement sensor 167.

[0086] 11C is a flowchart showing the operation of a modified example of the third embodiment. When user 1 performs voice input from microphone 121, or when an AR image such as that shown in material and container designation UI 133 is displayed on display 102 and user 1 performs a gesture motion to designate a material and a container (S130: Yes), processor 101 displays material recording information call UI 135a. When "Yes" is selected in material recording information call UI 135a (S151: Yes), processor 101 displays material recording information UI 135b. Here, the user selects the recording to use (S152).

[0087] Processor 101 compares the required amount with the remaining amount for all materials to be used, and if the required amount of at least one type of material is greater than the remaining amount (in other words, if the remaining amount of one or more types of material is insufficient) (S153: Yes), it notifies the user of the shortage of materials (S154). The notification method may be the generation of a sound effect or the display of material shortage notification UI 136a. Next, it also displays material shortage display UI 136b, which indicates the shortage amount of each material, etc.

[0088] When the user adds ingredients (S155: Yes) and inputs the amount of ingredients to be added (S156), processor 101 compares the required amount with the total amount obtained by adding the remaining amount to the additional amount for all ingredients to be used (S157). At this time, the UI 136b for displaying the shortage amount of ingredients is replaced with an UI 137b for displaying the additional amount of each ingredient. If the required amount of at least one ingredient to be used is greater than the total amount (in other words, if the total amount obtained by adding the remaining amount to the additional amount for one or more ingredients is still insufficient) (S157: Yes), or if the user does not add ingredients (S155: No), processor 101 uses the ingredient that is most deficient as a reference and calculates the required amounts and volumes of the other ingredients based on the recorded required amount ratios (S158). Thereafter, and if the required amount of all materials to be used is less than the total amount (S157: No) (in other words, if the required amount of all materials is sufficient), the user selects a container (S136), and after determining whether the three-dimensional shape of the container can be acquired (S137), executes process D, which leads to recording of the materials (S141) as necessary. If the user wishes to continue, the image capturing (S101) and distance measurement (S102) are repeated (S114: No), and if the user does not wish to continue, the process is terminated (S114: Yes).

[0089] If the material and container are not specified (S130: No), the density of the target object is acquired (S109), and process B is executed, leading to the display of the object type, mass, volume, and reference marker (S124). If the process continues, image capture (S101) and distance measurement (S102) are repeated (S114: No), and if the process does not continue, the process is terminated (S114: Yes). Note that, if the three-dimensional shape of the target object cannot be acquired in process B (S110: No), a new image capture direction is notified (S115), and image capture (S101) and distance measurement (S102) are performed in the new image capture direction.

[0090] If "No" is selected in the material recording information call UI 135a (S151: No), the user selects a material (S131), obtains the material density (S132), and if the three-dimensional shape of the material can be obtained (S133: Yes), the user inputs the required amount of material (S134), and the required volume is calculated (S135). This is followed by process C, followed by process D. If the three-dimensional shape of the material cannot be obtained in processes C and D (S133: No), or if the three-dimensional shape of the container cannot be obtained (S137: No), a new imaging direction is notified (S115), and the process returns to imaging (S101) and ranging (S102).

[0091] Furthermore, in step S153, if it is determined that the required amount of all materials to be used is less than or equal to the remaining amount (S153: No), there is a required amount of all materials, and therefore no additional material is required. Therefore, the user selects a container (S136), and after determining whether the three-dimensional shape of the container can be acquired (S137), process D is executed, leading to recording of the materials (S141) as necessary. If the user wishes to continue, the image capturing (S101) and distance measurement (S102) are repeated (S114: No); if the user does not wish to continue, the process is terminated (S114: Yes). If the three-dimensional shape of the container cannot be acquired in process D (S137: No), a new image capturing direction is notified (S115), and the process returns to image capturing (S101) and distance measurement (S102).

[0092] According to this embodiment, the weighing results of ingredients can be stored. When weighing a new ingredient, past weighing results can be referenced to know the required amount of each ingredient, the ratio of the required amount, and the current remaining amount. Furthermore, if there is a variation in the remaining amounts, the required amount of other ingredients can be measured according to the ingredient with the smallest amount, improving usability.

[0093] <Other embodiments> 12A and 12B are diagrams showing an example of the configuration of an augmented reality display system, and FIGS. 13A and 13B are diagrams showing an example of information output from the HMD 100. In FIG.

[0094] 12A is configured by connecting the HMD 100 and a server 510 via a communication network 520. The server 510 may perform analysis of the density and type information, as well as voice recognition processing of input data from the microphone 121 of the HMD 100, displaying an AR image such as that shown in the character input UI 131 on the display 102, and recognizing a gesture movement of the user 1 using the out-camera 111 or the distance measurement sensor 167, and the server 510 may then receive the results of the analysis.

[0095] Furthermore, the target object may be specified by displaying an AR image on the display 102 and recognizing a gesture (see FIG. 13A).

[0096] 12B, an HMD 100 (corresponding to a first augmented reality display device) and a smartphone 530 (corresponding to a second augmented reality display device) are communicatively connected and linked via a communication network 520. The processor 101 (corresponding to a first processor) of the HMD 100 combines an AR image displayed on a display 102 (corresponding to a first display) of the HMD 100 with a captured image of a real object captured by the outer camera 111 to create a composite image, which is then transmitted from a wireless LAN communicator 141 (corresponding to a first communicator) of the HMD 100A to the smartphone 530. If the display 102 is a see-through display, the captured image of the real object is not displayed on the display 102, but the processor 101 may perform image composition processing by superimposing an AR image on the captured image. If the display 102 is a non-transparent display, the image captured by the outer camera 111 is displayed on the display 102, and the AR image is also displayed on the display 102, so the processor 101 simply transmits the image displayed on the display 102 to the smartphone 530A as a composite image.

[0097] The smartphone 530 receives the composite image data via the wireless LAN communicator (corresponding to the second communicator) of the smartphone 530, and the processor (corresponding to the second processor) of the smartphone 530 displays it on the display 531 (corresponding to the second display) (see Figure 13B).

[0098] FIG. 14 is a diagram showing an example of an augmented reality display system in which a plurality of augmented reality display devices and smartphones 530A and 530B are linked to HMDs 100A and 100B, respectively.

[0099] The smartphones 530A and 530B and the HMDs 100A and 100B are connected to each other for communication. The HMDs 100A and 100B display AR images showing the same target object and its measurement results on the smartphones 530A and 530B, respectively, but the HMDs 100A and 100B display the AR images in different ways.

[0100] Therefore, the display image of the HMD 100A may be displayed from the smartphone 530A to the smartphone 530B, or the display image of the HMD 100B may be displayed from the smartphone 530B to the smartphone 530A. Here, for example, when the user 1 of the HMD 100A issues an instruction to switch the display destination of the composite image of the HMD 100A from the smartphone 530A to the smartphone 530B, the composite image of the HMD 100A is transmitted from the wireless LAN communicator 141 of the HMD 100A to the smartphone 530B, and transmission of the composite image to the smartphone 530A is stopped.

[0101] 15A and 15B are diagrams showing examples of AR image display when there are multiple target objects. In Fig. 15A, an AR image is displayed in which the measurement result field 220 of the target object 200 and the measurement result field 222 of the target object 202 correspond one-to-one.

[0102] In FIG. 15B, the measurement results of the target objects 200 and 202 are displayed together in a single integrated measurement result field 220A.

[0103] Whether to display the individual measurement result fields 220, 222 or to use the integrated measurement result field 220A may be specified by an input operation by the user 1, or the display control unit 18 of the HMD 100 may control switching between displaying multiple individual measurement result fields or displaying the integrated measurement result field 220A depending on the size of a blank area on the display 102 where the target objects 200, 202 are not displayed. The integrated measurement result field 220A displays the measurement results in the order in which the target objects 200, 202 were selected.

[0104] FIG. 16 illustrates an example of an AR display in a situation where multiple overlapping target objects are visible. Even when two or more types of target objects 206, 207, 208, and 209 are simultaneously identified, the HMD 100 may detect shape information, separate the approximate masses of the multiple target objects 206, 207, 208, and 209 that are randomly mixed within a specified range, and simultaneously display multiple measurement result fields 226, 227, 228, and 229 in parallel. In this case, the masses and volumes of similar objects or selected objects may be added together and displayed. For added convenience, target objects determined to be similar may be outlined in the same color to clearly indicate which objects are added together. Alternatively, multiple measurement results may be displayed in a single AR display area using the integrated measurement result field described above.

[0105] The present invention is not limited to the above-described embodiment, and modifications that do not deviate from the spirit of the present invention are included in the technical scope of the present invention.

[0106] For example, the augmented reality display device is implemented in a head-mounted display, but may also be implemented in smart glasses. In this case, the execution entity that executes the weighing assistance program, such as a processor, may be another information processing device, such as a smartphone.

[0107] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0108] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0109] The functions of the present invention described above may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Alternatively, they may be realized by software, such as by a microprocessor unit interpreting and executing an operating program that realizes each function. Hardware and software may also be used in combination.

[0110] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0111] 1: User 11: Range image data acquisition unit 12: Range image data storage unit 13: Object detection unit 14: Target object identification unit 15: Type estimation section 16: Type dictionary storage section 17:Physical quantity estimation section 17a: Density acquisition part 17b: 3D shape acquisition section 17c: Mass calculation section 18: Display control section 19: Communication control section 20: Density dictionary storage unit 100, 100A, 100B: HMD 101: Processor 102: Display 103:ROM 104: RAM 105: Storage 106: Bus 111: Rear camera 112: In-camera 121:Mike 122: Speaker 130: Operation button 131: Text input UI 132a, 132b, 132c: Reference manufacturer display switching UI 133: Material and container designation UI 134: UI for selecting whether to record material information 135a: Material recording information call UI 135b: Material Recording Information UI 136a: Material shortage notification UI 136b: Material shortage display UI 137a: UI for inputting whether or not to add materials 137b: Material addition amount display UI 141:Wireless LAN communication device 142: Near Field Communication Device 143:Telephone network communication device 150: Expansion I / F 160: Sensor group 162: Gyro sensor 163: Geomagnetic sensor 164: Acceleration sensor 165: Illuminance sensor 166: Proximity sensor 167: Distance sensor 168: Line of sight detection sensor 180: Battery 200, 201, 202, 203, 204, 206, 207, 208, 209: Target object 205: Real Objects 210, 211, 212, 213, 214, 216, 217: Frame 220A: Integrated measurement result column 220a, 222a: Required amount column 221, 222, 223, 224, 226, 226a, 226b, 227, 227a, 227b, 228, 229: Weighing result column 230, 232, 236a, 236b, 236c, 237a, 237b, 238a, 238b, 239a, 239b: Guide markers 250: Rescan notification 300: Akadama soil 301:Liquid fertilizer 310: Frame 311: Frame 320, 321, 322: Required amount field 330: Planter 331: Spoon 332: Cup 340, 341, 342a, 342b: Guide markers 500, 501: Augmented reality display system 510: Server 520: Communication Network 530, 530A, 530B: Smartphone 531: Display

Claims

1. A display control method for a display device, comprising: detecting a type of a real object included in a captured image generated by a camera of the display device; measuring a distance to each of a plurality of measurement points on the real object; recognizing a three-dimensional shape of the physical object based on the distances to each of the measurement points; estimating a volume of the physical object based on a three-dimensional shape of the physical object; receiving an input of a required mass, which is a required mass of the real object; calculating a required volume corresponding to the received required mass based on a density corresponding to the type of the detected real object; calculating a region of the three-dimensional shape of the physical object according to the calculated required volume based on the estimated volume of the physical object; a step of displaying a reference marker image indicating the calculated area on a display of the display device so as to be superimposed on the physical object; A display control method comprising:

2. 2. The display control method according to claim 1, The display control method, wherein the captured image is an image of an external scene captured by the camera of the display device.

3. 2. The display control method according to claim 1, the display device is a head-mounted display or smart glasses, A display control method, wherein the captured image is an image captured by the camera of the field of view of a user wearing the display device.

4. 4. A display control method according to claim 1, further comprising: calculating a mass of the physical object based on a density corresponding to the type of the detected physical object and the estimated volume of the physical object; displaying an image indicating the mass of the real object on the display; A display control method comprising:

5. 4. A display control method according to claim 1, further comprising: accepting an input of a plurality of required masses; displaying a plurality of reference marker images indicating the plurality of required masses on the display so as to be superimposed on the real object; A display control method comprising:

6. 6. A display control method according to claim 5, further displaying an augmented reality image showing a weighing result field indicating the plurality of required masses on the display; A display control method comprising:

7. 4. A display control method according to claim 1, further comprising: When receiving an input of a display position switching instruction, an image showing a new reference marker different from the reference marker image is displayed on the display so as to be superimposed on the real object; A display control method comprising:

8. 4. A display control method according to claim 1, further comprising: a step of notifying a user of the display device to take an image of the physical object from a different angle when the three-dimensional shape of the physical object cannot be recognized based on the distances to each of the measurement points; A display control method comprising:

Images