Display control method
The HMD-based display control method addresses inaccuracies in object measurement by using gaze detection and 3D shape reconstruction to accurately calculate and display physical quantities, improving measurement accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- MAXELL LTD
- Filing Date
- 2025-08-08
- Publication Date
- 2026-07-02
Smart Images
Figure 0007884126000001 
Figure 0007884126000002 
Figure 0007884126000003
Abstract
Description
Technical Field
[0001] The present invention relates to a display control method.
Background Art
[0002] In Patent Document 1, as an example of an information processing apparatus having a metering assist function, "an estimation unit that estimates the usage amount of at least one of the food ingredients to be cooked and the seasonings used for cooking based on the signal detected by the sensor, an index calculation unit that calculates a predetermined cooking index according to the estimation result by the estimation unit, and a notification control unit that controls to notify the cooking index calculated by the index calculation unit" is disclosed (abstract excerpt).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In the above Patent Document 1, in order to identify an object and estimate its mass based on the information obtained by imaging, an odor sensor or various concentration sensors (salt sensor or sugar sensor) installed in an external cooking appliance, for example, when a plurality of different objects have the same color and no difference is recognized from the information detected by the sensor, each object cannot be discriminated. In addition, it is necessary to separately prepare an external device in which the sensor is installed.
[0005] In addition, since the information obtained by imaging is two-dimensional information, the size of the object cannot be accurately grasped, and the estimated value of the mass may deviate greatly from the actual value.
[0006] Therefore, an object of the present invention is to provide a display control method capable of more preferably calculating and notifying the physical quantity of an object. [Means for solving the problem]
[0007] To solve the above problems, the present invention comprises the configuration described in the claims. To give one example, the present invention is a display control method in a display device, The processor provided in the display device detects the user's gaze using the gaze detection sensor, and based on the results, The display device Equipped for Real objects contained in images captured by the camera If it is determined that the user's gaze remains on the object for a certain period of time or longer, the system recognizes that the object the user's gaze remained on has been selected as the object to be measured for physical quantities, and Note subject A step of detecting the type of object, before Note subject For each of the multiple measurement points on the object, the step of measuring the distance to each measurement point, before Based on the distance to each of the measurement points, subject Steps to recognize the 3D shape of an object, before Note subject Based on the three-dimensional shape of the object, subject The steps involve estimating the volume of an object, before Note subject A step of receiving input for the required mass, which is the required mass of the object, before Note subject A step of calculating the required volume corresponding to the received required mass based on the density corresponding to the type of object, before Estimated subject Based on the volume of the object, subject The steps include: calculating a region in the three-dimensional shape of an object that corresponds to the required volume calculated above; before The above refers to the reference marker image that indicates the calculated area. subject The display device superimposed on the object Equipped for The steps to be displayed on the screen, include . [Effects of the Invention]
[0008] According to the present invention, a display control method can be provided that can more preferably calculate and notify the physical quantities of an object. Other problems, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]
[0009] [Figure 1] Figure showing the outline of the augmented reality display device according to the first embodiment. [Figure 2] Hardware configuration diagram of the HMD. [Figure 3] Block diagram showing an example of the configuration of the measurement assist program executed by the processor of the HMD. [Figure 4] Flowchart showing the operation of the HMD according to the first embodiment. [Figure 5] Flowchart showing the operation of the HMD according to the first embodiment. [Figure 6] Figure showing an example of the display of the rescan notification. [Figure 7] Figure showing the outline of the augmented reality display device according to the second embodiment. [Figure 8] Flowchart showing the operation of the HMD according to the second embodiment. [Figure 9] Figure showing an example of switching the display position of the reference marker. [Figure 10] Figure showing the outline of the augmented reality display device according to the third embodiment. [Figure 11A] Figure showing the outline of the augmented reality display device according to the third embodiment. [Figure 11B] Figure showing an example of the screen display used in the modification of the third embodiment. [Figure 11C] Flowchart showing the operation of the modification example of the third embodiment. [[ID=�9]] [Figure 12A] Figure showing a configuration example of the augmented reality display system. [Figure 12B] Figure showing a configuration example of the augmented reality display system. [Figure 13A] Figure showing an example of information output from the HMD. [Figure 13B] Figure showing an example of information output from the HMD. [Figure 14] Figure showing an example of the augmented reality display system. [Figure 15A] Figure showing an example of AR image display when there are multiple target objects. [Figure 15B] Figure showing an example of AR image display when there are multiple target objects. [Figure 16]This diagram shows an example of AR display in a situation where multiple objects overlap and are visible. [Modes for carrying out the invention]
[0010] This invention contributes to Sustainable Development Goal 8.2 of the United Nations (Increase economic productivity through diversification, technological advancement, and innovation, particularly in industries that enhance the value of goods and services, and in labor-intensive industries), for example, by enabling diversification and technological improvement in labor-intensive industries. Examples of embodiments of this invention will be described below with reference to the drawings. The same reference numerals are used for identical components throughout the drawings, and redundant explanations are omitted.
[0011] <First Embodiment> The first embodiment is an embodiment in which a head-mounted display is used as an augmented reality display device, and a physical quantity of a real object (at least one of volume or mass) is displayed on the display as an augmented reality image using a real object weighing assistance function.
[0012] Figure 1 shows an overview of the augmented reality display device according to the first embodiment.
[0013] The HMD100 in Figure 1 is equipped with a transparent display 102. User 1, wearing the HMD100, views real objects seen through the display 102, overlaying the augmented reality image (hereinafter referred to as "AR image") displayed on the display 102.
[0014] When the HMD100 recognizes target objects 200, 201, 202, 203, and 204 from the real-world objects captured by the out-camera 111 mounted on the HMD100, it displays frames 210, 211, 212, 213, and 214 surrounding the target objects.
[0015] The HMD100 then estimates the physical quantities of each target object 200, 201, 202, 203, and 204, and displays the estimation results in AR images on the display 102, shown in the measurement result columns 220, 221, 222, 223, and 224. At that time, the measurement result columns 220, 221, 222, 223, and 224 are displayed on the display 102 in the vicinity of the target objects 200, 201, 202, 203, and 204.
[0016] Each measurement result column (220, 221, 222, 223, 224) displays the name of the object and its physical quantity. The physical quantity here may be the volume of the object, or, if the density of the object is known, the mass calculated from the density and volume. In this embodiment, mass and volume are used as the physical quantities.
[0017] In addition to the target objects 200, 201, 202, 203, and 204, the images captured by the rear camera 111 also include a real object 205. However, since the HMD 100 does not recognize the real object 205 as a target object, no frame is added to it, and therefore, physical quantity estimation and result display are not performed.
[0018] If the HMD100 wants to correct the results of recognizing the types of target objects 200, 201, 202, 203, and 204 from the images captured by the rear camera 111, the HMD100 may be configured to allow user 1 to input voice information through the microphone 121, or to display an AR image as shown in the text input UI 131 on the display 102 and input text information by having the rear camera 111 and distance sensor 167 recognize the gesture movements of user 1.
[0019] Figure 2 is a hardware configuration diagram of the HMD100.
[0020] The HMD100 is comprised of a processor 101, a display 102, a ROM 103, a RAM 104, storage 105, an outward camera 111, an inward camera 112, a microphone 121, a speaker 122, operation buttons 130, a wireless LAN communicator 141, a proximity wireless communicator 142, a telephone network communicator 143, an expansion I / F 150, a sensor group 160, and a battery 180, all interconnected via a bus 106.
[0021] The sensor group 160 may include a GPS (Global Positioning System) 161, a gyroscope 162, a geomagnetic sensor 163, an accelerometer 164, an illuminance sensor 165, a proximity sensor 166, a distance sensor 167, and a gaze detection sensor 168.
[0022] The distance measuring sensor 167 can be a ToF (Time Of Flight) sensor or an ultrasonic sensor. In addition, if the rear camera 111 is a stereo camera, the distance to a real object can be measured using the parallax between the left and right cameras, so the rear camera 111 can also be used as the distance measuring sensor 167. If the rear camera 111 is a ToF camera, the rear camera 111 can also be used as the distance measuring sensor 167.
[0023] The ROM 103 or storage 105 stores the weighing assist program for the HDM 100.
[0024] Figure 3 is a block diagram showing an example of the configuration of a weighing assist program executed by the processor 101 of the HMD100.
[0025] The weighing assist program includes a distance image data acquisition unit 11, a distance image data storage unit 12, an object detection unit 13, a target object identification unit 14, a type estimation unit 15, a type dictionary storage unit 16, a physical quantity estimation unit 17, a display control unit 18, a communication control unit 19, and a density dictionary storage unit 20. The physical quantity estimation unit 17 includes a density acquisition unit 17a, a 3D shape acquisition unit 17b, and a mass calculation unit 17c. The processor 101 loads the weighing assist program into the RAM 104 and executes it, thereby realizing the functions of each of the above units. The processor 101 may be composed of an integrated circuit that realizes the same functions as the weighing assist program. Details of the processing of each of the above units will be explained with reference to the flowcharts in Figure 4 and below.
[0026] Alternatively, a weighing assist program may be executed on the HMD100 and a server or information processing device, and the results may be returned to the HMD100 and displayed on the display 102.
[0027] The operation of the HMD100 according to the first embodiment will be described with reference to Figures 4 to 6. Figures 4 and 5 are flowcharts showing the operation of the HMD100 according to the first embodiment. Figure 6 is a diagram showing an example of the display of a rescan notification.
[0028] When the main power of the HMD100 is turned ON, the flowchart in Figure 4 begins. The rear camera 111 captures the external scenery and outputs the captured image to the distance image data acquisition unit 11 (S101).
[0029] The distance measuring sensor 167 has a distance measuring area that includes the field of view of the rear camera 111. The distance measuring sensor 167 measures the distance in synchronization with the rear camera 111 and outputs the distance measurement data to the distance image data acquisition unit 11 (S101). The distance image data acquisition unit 11 stores the distance image data, which associates the imaging data and the distance data, in the distance image data storage unit 12 (S102).
[0030] The object detection unit 13 reads distance image data from the distance image data storage unit 12. The object detection unit 13 performs subject detection processing on the distance image data and detects real objects (subjects) captured in the captured image (S103).
[0031] When the object detection unit 13 detects at least one subject (S103:Yes) and user 1 selects a real object to be measured for physical quantities ("target object") (S104:Yes), the type estimation unit 15 estimates the type of the target object (S105).
[0032] As an example of a method for selecting a target object, we will describe an embodiment using a gaze detection sensor 168. The gaze detection sensor 168 acquires the face image of user 1 captured by the in-camera 112 and detects the gaze from the area in which the eyes are captured. The target object identification unit 14 recognizes that a real object detected by the object detection unit 13 has been selected as a target object if the gaze remains on that real object for a certain period of time or longer.
[0033] As another example of the selection method, the object detection unit 13 displays multiple frames 210, 211, 212, 213, and 214 on the display 102, each containing the region in which a real object detected from the imaging data is captured. When user 1 performs a gesture to specify one of the frames 210, 211, 212, 213, or 214, the gesture is captured in the image taken by the rear camera 111. The target object identification unit 14 then analyzes the gesture based on the distance image data to recognize whether a target object has been selected. When input is made using gestures, the rear camera 111 is an example of an information input device.
[0034] Alternatively, the object identification unit 14 may recognize the audio data collected by the microphone 121 and determine whether a target object has been selected. Therefore, the microphone 121 is an example of an information input device.
[0035] If the object detection unit 13 does not detect any subject (S103: No), or if the target object identification unit 14 does not accept the selection of a target object (S104: No), and the weighing assist function of the HMD 100 is to be used continuously (S114: No, see Figure 5), new captured images and distance measurement data are acquired (S101, S102).
[0036] The type estimation unit 15 extracts image features such as the shape and color of the target object and compares them with the image features of various real objects registered in the type dictionary storage unit 16 to estimate the type of the target object. The type dictionary may be stored on a server connected to the HMD 100 via communication, and the HMD 100 may send a request for type determination to the server as needed and receive the result.
[0037] Alternatively, the type estimation unit 15 may recognize characters, figures, and symbols written on the surface of the target object or on packages placed around the target object, and estimate the type of the target object.
[0038] The type estimation unit 15 notifies the user 1 of the estimated type of the object by displaying it on the display 102 or by outputting sound from the speaker 122. When the user 1 approves the estimation result (S106: Yes), the type of the object is determined (S108).
[0039] If User 1 does not approve the estimation result (S106: No), User 1 inputs correction information for the type (S107). For example, if the type estimation unit 15 estimated the type to be "sugar," User 1 may want to correct it to "granulated sugar" or "salt." Input of correction information may be accepted by having the microphone 121 collect the voice spoken by User 1 and the type estimation unit 15 perform voice analysis processing, or by displaying an AR image as shown in the character input UI 131 on the display 102, recognizing User 1's gesture movements using the out camera 111 and distance sensor 167, and then having the type estimation unit 15 analyze the input character information. Once the input of correction / supplementary information is accepted, the type of the target object is determined (S108).
[0040] The processes described above, from step S101 to step S108, constitute the process of determining the type of the target object (referred to as process "A" in Figure 4).
[0041] Once the type of target object is determined (S108), the density acquisition unit 17a of the physical quantity estimation unit 17 acquires the density of that type (S109). The density may be obtained by referring to a density dictionary stored in a server with which the HMD 100 is connected, or the HMD 100's storage 105 may be pre-equipped with a density dictionary storage unit 20.
[0042] The type dictionary and density dictionary may be prepared according to the scenario in which the HMD100 is used. For example, when the HMD100 is used as a cooking aid, it is expected that the measurement of various seasonings (e.g., soy sauce, sauce, butter, margarine, vegetable oil, sugar, salt, Chinese seasonings, etc.), as well as flour, breadcrumbs, and water will be necessary. Therefore, a dictionary containing the image features (color, shape, etc.) and density of these ingredients may be prepared as a cooking dictionary. It is also preferable that the image features and density of various vegetables, meats, and processed foods (tofu, etc.) as cooking ingredients are also stored.
[0043] Furthermore, in scenarios where the HMD100 is used to assist with gardening tasks, it is preferable that the image features and densities of soil types, such as black soil, akadama soil, kanuma soil, and leaf mold, are stored in the dictionary.
[0044] Furthermore, for example in plastering work, it is preferable to have a dictionary containing image features and densities of real-world objects used in various situations where measurement is required, such as various types of plaster wall materials, water, cement, etc.
[0045] If the accuracy of the HMD100's estimation of physical quantities is sufficiently high, it may be used, for example, in a dispensing pharmacy to weigh various bases (e.g., petrolatum, macrogol, Plastibase, etc.) and drugs mixed with them, or in the compounding of drugs in research facilities.
[0046] Next, the 3D shape acquisition unit 17b of the physical quantity estimation unit 17 reads distance image data from the distance image data storage unit 12 and acquires the 3D shape of the target object (S110). The distance measuring sensor 167 may measure the distance from the distance measuring sensor 167 (HMD100) to each of the multiple measurement points on the target object, interpolate the connecting lines that connect adjacent measurement points, and reconstruct the surface including the connecting lines to acquire the 3D shape. Alternatively, surface rendering may be performed from the coordinates of the measurement points to acquire the 3D shape of the target object. Acquisition of the 3D shape does not require acquiring the complete 3D shape of the target object, and the acquisition level may be adjusted according to the intended use of the HMD100.
[0047] The 3D shape acquisition unit 17b may determine whether or not it can acquire a 3D shape if, for example, imaging distance measurement data of the target object is obtained from only one direction (e.g., the distance in the depth direction of the measurement point is within the threshold for determining the loss of depth information, for example, 1 cm), and may determine that it is impossible to acquire a 3D shape if imaging distance measurement data is obtained from at least two directions, preferably three or more directions including the front, back, and depth directions of the target object.
[0048] If the 3D shape acquisition unit 17b can reconstruct the approximate 3D shape of the target object (S110: Yes), the volume / mass calculation unit 17c calculates the volume of the 3D shape (S111), and then multiplies it by the density to calculate the mass (S112).
[0049] The volume / mass calculation unit 17c outputs the type of object and its physical quantity to the display control unit 18, which generates an AR image displaying the type of object and its physical quantity in the weighing result column and displays it on the display 102 (S113). If the weighing assist function by the HMD 100 is to be continued (S114: No), the process returns to step S101.
[0050] On the other hand, if the 3D shape acquisition unit 17b determines that it cannot acquire the approximate 3D shape of the target object (S110: No), it issues a rescan notification 250 to user 1 prompting it to acquire distance image data of the target object from a different angle as shown in Figure 6 (S115), and returns to step S101.
[0051] According to this embodiment, the shape of the object to be measured is measured in three dimensions and its volume is determined by the output from the out-camera 111 and distance sensor 167 mounted on the HMD100. Furthermore, if the density is known, the mass of the object is determined using the volume and density. As a result, the HMD100 alone can accurately calculate the physical quantities of the object (volume and the mass calculated based on it) and notify the user 1.
[0052] <Second Embodiment> The second embodiment is an embodiment that, in addition to the first embodiment, displays the information in a way that makes it appear as if guide markers have been drawn.
[0053] Figure 7 shows an overview of the augmented reality display device according to the second embodiment. For each of the target objects 200 and 202 shown in Figure 7, multiple guide markers 230 and 232 are displayed in AR, indicating the required amount specified by user 1 for each of the target objects 200 and 202, rather than the total amount of the target object 200 and 202. The object type and the mass and volume corresponding to the specified required amount are displayed in the required amount fields 220a and 222a. Frames 210 and 212 are added to the target objects 200 and 202.
[0054] Furthermore, frames 216 and 217 are displayed in AR for each of the target objects 206 and 207. Multiple required quantities are specified for each of the target objects 206 and 207. Therefore, for target object 206, guide markers 236a and 236b, and the physical quantities corresponding to each guide marker are displayed in measurement result fields 226a and 226b. Similarly, for target object 207, guide markers 237a and 237b, and the physical quantities corresponding to each guide marker are displayed in measurement result fields 227a and 227b.
[0055] Figure 8 is a flowchart showing the operation of the HMD100 according to the second embodiment. In the second embodiment, steps S120 to S124 are added to the flowchart of the first embodiment.
[0056] When the density acquisition unit 17a acquires the density of the target object (S109), the 3D shape acquisition unit 17b acquires the 3D shape of the target object (S110: Yes). When user 1 specifies the required amount of the target object (S120: Yes), user 1 either inputs voice via microphone 121 or displays an AR image as shown in the text input UI 131 on display 102, and inputs the required amount of the target object by recognizing user 1's gesture movements using the rear camera 111 and distance sensor 167 (S121).
[0057] The volume / mass calculation unit 17c calculates the required volume of the object from its density, three-dimensional shape, and required quantity (S122).
[0058] The volume / mass calculation unit 17c calculates the position of the guide marker to be displayed on the object based on the object's three-dimensional shape and required volume (S123).
[0059] The volume / mass calculation unit 17c outputs the position of the reference marker, the type of the target object, the mass corresponding to the required amount, and the volume to the display control unit 18. The reference marker is then superimposed on the real object in AR display, and the weighing result fields 226a, 226b, 227a, and 227b are displayed in AR near the real object (S124). The process then proceeds to step S114.
[0060] On the other hand, if User 1 does not specify the required amount of the target object (S120: No), the process proceeds from step S111 to S114, as in the first embodiment.
[0061] According to this embodiment, when it is necessary to measure a required amount less than the total amount of the target object, the target object can be divided into the required amount by dividing it along the guide markers displayed on the target object using AR.
[0062] Figure 9 shows an example of switching the display position of a reference marker. The area to which the reference marker is displayed may also be changed. For example, if a reference marker 236b is displayed in the left area of the target object 206 (e.g., an apple slice) in the figure, if a reference marker display switching operation is performed in step S124 or later, the volume / mass calculation unit 17c may execute step S123 again, and the display control unit 18 may be configured to display a new reference marker 236c in an area different from the superimposed display area of the initially displayed reference marker.
[0063] Similarly, while the initial display shows a guide marker 238a by dividing the internal space of the target object 208 (e.g., a planter) vertically, the system may be configured to display a new guide marker 238b by switching the display of the guide markers, which divides the internal space along the depth direction.
[0064] Furthermore, while the initial display shows a guide marker 239a that is kept horizontal with the target object 209 (e.g., a glass) upright, the system may be configured to display a guide marker 239b that is kept horizontal with the target object 209 tilted.
[0065] The type of guide marker to display can be determined by user 1 specifying the type of guide marker via voice input from microphone 121, or by the type estimation unit 15 determining the attributes of the target object and the volume / mass calculation unit 17c automatically selecting a guide marker corresponding to those attributes. For example, if the attribute of the target object is determined to be "solid: solid," a guide marker along the outer surface of the target object may be automatically selected; if it is determined to be "solid: container," a guide marker along the containment space of the container may be automatically selected; and if it is determined to be "liquid," a guide marker with a horizontal water surface and an outline along the inner surface of the container may be automatically selected. The display switching of the guide marker may also be performed by displaying AR images as shown in the guide marker display switching UI 132a, 132b, and 132c on the display 102 and recognizing the user 1's gesture movements using the out camera 111 and distance sensor 167.
[0066] According to this embodiment, by specifying the required amount for the target object and attaching a guide marker, only the required amount can be extracted.
[0067] <Third Embodiment> The third embodiment, in addition to the second embodiment, is an embodiment in which, when there is a container and a material to be placed in the container (an example of a target object), the shape information of the material and the shape information of the container (an example of a target object) are detected, the volume at which the mass of the material is approximately a certain value and the corresponding volume inside the container (space) are determined, and a guide marker is attached to the container so that this can be seen.
[0068] Figure 10 shows an overview of an augmented reality display device according to the third embodiment. In Figure 10, for example, there is akadama soil 300 as a material and a planter 330 as a container. A frame 310 indicating that the akadama soil 300 has been recognized as a material is displayed in AR. The required amount of akadama soil 300 entered by user 1 is displayed in the required amount field 320, showing the type "akadama soil", the value of the required amount converted to mass, and the value converted to volume. A guide marker 340 indicating how deep the planter 330 would be if the required amount of akadama soil 300 were added is superimposed and displayed in AR.
[0069] The material can be a solid or a liquid. For example, it could be liquid fertilizer 301 as the material and a spoon 331 as the container. A frame 311 is displayed in AR on the liquid fertilizer 301 to indicate that it has been recognized as a material. The required amount field 321 for the liquid fertilizer 301 displays the type of material, the required amount converted to mass, and the required amount converted to volume. A guide marker 341 is superimposed in AR on the spoon 331 to indicate how deep it would be if the required amount of liquid fertilizer 301 were added.
[0070] The required amount field 322 shows that two required amounts of liquid fertilizer 301 have been entered. The cup 332, which serves as the container, displays guide markers 342a and 342b corresponding to the two required amounts. The display switching of these guide markers may also be performed by displaying an AR image, as shown in the text input UI 131, on the display 102 and recognizing the user 1's gestures using the rear camera 111 and distance sensor 167. Furthermore, when storing the weighing results, the material and container specification UI 133 and the material information recording requirement selection UI 134 may be displayed as AR images to accept input operations.
[0071] Figure 11A is a flowchart showing the operation of the HMD100 according to the third embodiment. In the third embodiment, steps S130 to S141 are added to the flowchart of the second embodiment.
[0072] When User 1 provides voice input through microphone 121, or displays an AR image such as the material and container specification UI 133 on display 102, and User 1 performs a gesture to input that they are specifying a material and container (S130: Yes), the object detection unit 13 places a frame around all detected objects. When User 1 selects a frame 310 to be selected as a material using a gesture (S131), the type estimation unit 15 estimates the type of material, and the density acquisition unit 17a acquires the density of the material (S132). If User 1 inputs the type of material using voice input, the type estimation unit 15 may estimate the type based on the input information. Frames placed around real objects that were not selected are hidden.
[0073] The 3D shape acquisition unit 17b acquires the 3D shape of the material (S133: Yes), and when user 1 inputs the required amount of material (S134), the volume / mass calculation unit 17c calculates the required volume of the target object from the density of the material, the 3D shape of the target object, and the required amount (S135).
[0074] Next, User 1 selects a container (S136). The container selection operation can be performed by User 1 using a gesture to point to a container, which the object detection unit 13 recognizes, or by selecting the material frame and the container frame separately in step S131; the type of selection operation is not limited.
[0075] When the 3D shape acquisition unit 17b acquires the 3D shape of the container (S137: Yes), the volume / mass calculation unit 17c calculates the position of the guide marker to be displayed on the container based on the required volume of the material and the shape of the container (S138).
[0076] The volume / mass calculation unit 17c outputs the position of the reference marker, the type of material, the mass corresponding to the required amount, and the volume to the display control unit 18. The reference marker is superimposed on the container and displayed in AR, and the required amount fields 320 and 321 are also displayed in AR near the container (S139).
[0077] The processor 101 displays a UI 133 for specifying materials and containers, and a UI 134 for selecting whether or not to record material information. If recording is required (S140: Yes), it records the type, required amount, required amount ratio, and remaining amount of one or more materials (S141).
[0078] After recording the remaining amount (S141), or if recording is not required (S140: No), or if you wish to continue (S114: No), the imaging (S101) and distance measurement (S102) are repeated. If you wish to finish (S114: Yes), the above series of processes is terminated.
[0079] If the 3D shape of the material cannot be obtained (S133: No), or if the 3D shape of the container cannot be obtained (S137: No), a new imaging direction is notified (S115), and the process returns to steps S101 and S102.
[0080] Furthermore, if User 1 does not specify a material and container (S130: No), the process proceeds to process B, which involves obtaining the density of the target object (S109) and displaying the object type, mass, volume, and reference marker (S124), similar to the second embodiment. If the process is to be continued (S114: No), imaging (S101) and distance measurement (S102) are repeated. If the process is to be terminated (S114: Yes), the above series of processes is terminated.
[0081] According to this embodiment, when the target object consists of a material and a container to hold it, a guideline marker can be displayed indicating when the required amount of material has been placed in the container.
[0082] Referring to Figures 11B and 11C, an example of processing when using the recorded information after recording the information of the target object (material) in step S141 will be explained.
[0083] Figure 11B shows an example of a screen display used in a variation of the third embodiment.
[0084] The material record information retrieval UI 135a shown in Figure 11B is a UI that confirms whether it is necessary to retrieve the record of previously used materials. When information requesting a retrieval is entered into the material record information retrieval UI 135a, the material record information UI 135b is displayed.
[0085] Furthermore, if the weighing results indicate a shortage of materials, a material shortage notification UI136a is displayed, followed by a material shortage amount display UI136b showing the amount of the missing material. Next, a material addition requirement input UI137a is displayed to confirm whether additional materials are needed, and if "Yes" is selected (S155:Yes), a material addition amount display UI137b is displayed, allowing user 1 to select the material to add and input the amount to be added (S156). Inputting the amount to be added may be configured so that user 1 inputs it by voice through the microphone 121 of the HMD 100, or by displaying an AR image as shown in the text input UI131 on the display 102 and recognizing user 1's gesture movements using the out camera 111 and distance sensor 167 to input text.
[0086] Figure 11C is a flowchart showing the operation of a modified example of the third embodiment. When User 1 makes a voice input from the microphone 121, or when an AR image such as the material and container specification UI 133 displayed on the display 102 is displayed, and User 1 makes a gesture to input that they will specify the material and container (S130: Yes), the processor 101 displays the material record information call UI 135a. When "Yes" is selected in the material record information call UI 135a (S151: Yes), the processor 101 displays the material record information UI 135b. Here, the user selects the record to use (S152).
[0087] The processor 101 compares the required amount with the remaining amount for all materials to be used, and if the required amount of at least one type of material is greater than the remaining amount (in other words, if the remaining amount of one or more types of material is insufficient) (S153: Yes), it notifies the user of the material shortage (S154). As a notification method, a sound effect or the display of the material shortage notification UI 136a may be used. Next, the material shortage amount display UI 136b, which shows the shortage amount of each material, is displayed.
[0088] When the user adds material (S155:Yes) and the user inputs the amount of material to be added (S156), the processor 101 compares the required amount with the remaining amount plus the added amount for all materials to be used (S157). At this time, the material shortage display UI 136b is replaced with the material addition amount display UI 137b, which shows the amount to be added for each material. If the required amount of at least one type of material is greater than the total amount (in other words, if the total amount of one or more types of materials plus the added amount is still insufficient) (S157:Yes), and if the user does not add material (S155:No), the processor 101 calculates the required amount and volume of the other materials based on the recorded required amount ratio, using the material that is most insufficient as the basis (S158). Subsequently, if the required amount of all materials used is less than or equal to the total amount (S157: No) (in other words, if the required amount of all materials is sufficient), the user selects a container (S136), determines whether the 3D shape of the container can be obtained (S137), and then executes process D, which includes recording the materials as needed (S141). If the process continues, imaging (S101) and distance measurement (S102) are repeated (S114: No); otherwise, the process ends (S114: Yes).
[0089] If no material or container is specified (S130:No), the density of the target object is acquired (S109), and process B is executed to display the object type, mass, volume, and reference marker (S124). If the process is to continue, imaging (S101) and distance measurement (S102) are repeated (S114:No); otherwise, the process is terminated (S114:Yes). If the 3D shape of the target object cannot be acquired in process B (S110:No), a new imaging direction is notified (S115), and imaging (S101) and distance measurement (S102) are performed in that new imaging direction.
[0090] If "No" is selected in the material recording information retrieval UI135a (S151: No), the user selects a material (S131), the material density is obtained (S132), and if the 3D shape of the material can be obtained (S133: Yes), the user inputs the required amount of material (S134), and the required volume is calculated (S135). Process C and the subsequent process D are then executed. If the 3D shape of the material cannot be obtained in processes C and D (S133: No), and if the 3D shape of the container cannot be obtained (S137: No), a new imaging direction is notified (S115), and the process returns to imaging (S101) and distance measurement (S102).
[0091] Furthermore, in step S153, if it is determined that the required amount of all materials to be used is less than or equal to the remaining amount (S153: No), then no additional materials are needed as there is sufficient quantity of all materials. In this case, the user selects a container (S136), and after determining whether the 3D shape of the container can be acquired (S137), process D is executed, which includes recording the materials as needed (S141). If the user wishes to continue, imaging (S101) and distance measurement (S102) are repeated (S114: No); otherwise, the process is terminated (S114: Yes). In process D, if the 3D shape of the container cannot be acquired (S137: No), a new imaging direction is notified (S115), and the process returns to imaging (S101) and distance measurement (S102).
[0092] According to this embodiment, the weighing results of materials can be stored. When weighing again, past weighing results can be referenced to determine the required amount, required amount ratio, and current remaining amount for each material. Furthermore, if there is variation in the remaining amounts, the required amounts of other materials can be measured to match the amount of the material with the least amount remaining, improving usability.
[0093] <Other Embodiments> Figures 12A and 12B show examples of the configuration of an augmented reality display system, while Figures 13A and 13B show examples of information output from the HMD100.
[0094] The augmented reality display system 500 in Figure 12A is configured by connecting the HMD 100 and the server 510 via a communication network 520. The system may also be configured to perform voice recognition processing of data such as density and type information, as well as voice recognition processing of data input from the microphone 121 of the HMD 100, display AR images as shown in the text input UI 131 on the display 102, and analyze text input information entered by recognizing the gesture movements of the user 1 using the out-camera 111 and distance sensor 167, with the server 510 receiving the results.
[0095] Alternatively, the target object may be specified by displaying it on the display 102 using AR and recognizing gesture movements (see Figure 13A).
[0096] The augmented reality display system 501 in Figure 12B communicates and cooperates with the HMD 100 (corresponding to the first augmented reality display device) and the smartphone 530 (corresponding to the second augmented reality display device) via a communication network 520. The processor 101 (corresponding to the first processor) of the HMD 100 combines the AR image displayed on the HMD 100's display 102 (corresponding to the first display) with the image of a real object captured by the rear camera 111 to create a composite image, which is then transmitted from the HMD 100A's wireless LAN communicator 141 (corresponding to the first communicator) to the smartphone 530. If the display 102 is a transparent display, the image of the real object is not displayed on the display 102, but the processor 101 can perform image synthesis processing by superimposing the AR image onto the captured image. If the display 102 is an opaque display, the image captured by the rear camera 111 is displayed on the display 102, and an AR image is also displayed on the display 102, so the processor 101 only needs to send the images displayed on the display 102 as a composite image to the smartphone 530A.
[0097] The smartphone 530 receives synthesized image data via its wireless LAN communication device (corresponding to the second communication device), and the smartphone 530's processor (corresponding to the second processor) displays it on the display 531 (corresponding to the second display) (see Figure 13B).
[0098] Figure 14 shows an example of an augmented reality display system in which multiple augmented reality display devices, smartphones 530A and 530B, are linked with HMDs 100A and 100B, respectively.
[0099] Smartphones 530A and 530B, and HMDs 100A and 100B are each connected to one another via communication. Smartphones 530A and 530B, respectively, display AR images of the same target object and its measurement results on HMDs 100A and 100B, but the way the AR images are displayed differs between HMD 100A and HMD 100B.
[0100] Therefore, the display image of HMD100A may be displayed from smartphone 530A to smartphone 530B, or the display image of HMD100B may be displayed from smartphone 530B to smartphone 530A. Here, for example, if user 1 of HMD100A gives an instruction to switch the display destination of the composite image of HMD100A from smartphone 530A to smartphone 530B, the wireless LAN communication device 141 of HMD100A transmits the composite image of HMD100A to smartphone 530B, and the transmission of the composite image to smartphone 530A is stopped.
[0101] Figures 15A and 15B show examples of AR image display when there are multiple target objects. In Figure 15A, an AR image is displayed in which the weighing result field 220 of target object 200 and the weighing result field 222 of target object 202 correspond in a 1:1 ratio.
[0102] In Figure 15B, the weighing results for the target objects 200 and 202 are displayed together in a single integrated weighing result field 220A.
[0103] The user 1 may specify whether to display individual weighing result fields 220 and 222 or use the integrated weighing result field 220A, or the display control unit 18 of the HMD 100 may switch between displaying multiple individual weighing result fields or the integrated weighing result field 220A depending on the size of the blank area on the display 102 where the target objects 200 and 202 are not displayed. The integrated weighing result field 220A displays the weighing results in the order in which the target objects 200 and 202 were selected.
[0104] Figure 16 shows an example of AR display in a situation where multiple objects overlap and are visible. Even when the HMD100 simultaneously identifies two or more objects 206, 207, 208, and 209 of different types, it can detect shape information and separate the approximate masses of the multiple objects 206, 207, 208, and 209 that are mixed together haphazardly within a specified range, and simultaneously display multiple weighing result fields 226, 227, 228, and 229 in parallel. In this case, the masses and volumes of similar objects or some of the selected objects may be added together and displayed. It would be even more convenient to indicate which objects have been added together, for example, by adding outlines of the same color to objects that have been determined to be similar. Alternatively, multiple weighing results may be displayed in a single AR display area using the integrated weighing result field described above.
[0105] The present invention is not limited to the embodiments described above, and modifications that do not depart from the spirit of the invention are included within the technical scope of the present invention.
[0106] For example, while the augmented reality display device is implemented in a head-mounted display, it could also be implemented in smart glasses. In that case, the execution entity that runs the metering assistance program, such as the processor, could be another information processing device, such as a smartphone.
[0107] Although embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.
[0108] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is executed may also be changed.
[0109] The functions of the present invention described above may be implemented in hardware, either partially or entirely, by designing them, for example, using an integrated circuit. Alternatively, they may be implemented in software by having a microprocessor unit or the like interpret and execute an operating program that implements each of these functions. Hardware and software may also be used in combination.
[0110] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is reasonable to assume that almost all components are interconnected. [Explanation of Symbols]
[0111] 1: User 11: Distance image data acquisition unit 12: Distance image data storage unit 13: Object detection unit 14: Target object identification unit 15: Type Estimation Unit 16: Type Dictionary Memory Unit 17:Physical quantity estimation section 17a: Density acquisition part 17b: 3D shape acquisition part 17c: Mass calculation section 18: Display Control Unit 19: Communication Control Unit 20: Density dictionary memory section 100, 100A, 100B: HMD 101: Processor 102: Display 103: ROM 104: RAM 105: Storage 106: Bus 111: Rear Camera 112: Front camera 121: Mike 122: Speaker 130: Operation Buttons 131: Text Input UI 132a, 132b, 132c: Reference manufacturer display switching UI 133: UI for specifying materials and containers 134: UI for selecting whether or not to record material information 135a: Material Record Information Retrieval UI 135b: Material Record Information UI 136a: Material shortage notification UI 136b: Material shortage display UI 137a: UI for inputting whether or not to add materials 137b: Material addition amount display UI 141:Wireless LAN communication device 142: Proximity radio transceiver 143:Telephone network communication device 150: Extended I / F 160: Sensor group 162: Gyroscope sensor 163: Geomagnetic sensor 164: Accelerometer 165: Illuminance sensor 166: Proximity sensor 167: Distance measuring sensor 168: Eye-tracking sensor 180: Battery 200, 201, 202, 203, 204, 206, 207, 208, 209: Target object 205: Real object 210, 211, 212, 213, 214, 216, 217: Frame 220A: Integrated weighing results column 220a, 222a: Required amount column 221, 222, 223, 224, 226, 226a, 226b, 227, 227a, 227b, 228, 229: Weighing result column 230, 232, 236a, 236b, 236c, 237a, 237b, 238a, 238b, 239a, 239b: Reference markers 250: Rescan Notification 300: Akadama soil 301:Liquid fertilizer 310: Frame 311: Frame 320, 321, 322: Required amount field 330: Planter 331: Spoon 332: Cup 340, 341, 342a, 342b: Reference markers 500, 501: Augmented reality display systems 510: Server 520: Communication Network 530, 530A, 530B: Smartphones 531: Display
Claims
1. A display control method for a display device, The processor provided in the aforementioned display device Based on the results of detecting the user's gaze on the display device using a gaze detection sensor, if it is determined that the user's gaze has remained on a real object included in the image captured by the camera provided on the display device for a certain period of time or longer, the system recognizes that the real object on which the gaze remained has been selected as the target object for measuring physical quantities. A step of detecting the type of the target object, The steps include measuring the distance to each of the multiple measurement points on the object, A step of recognizing the three-dimensional shape of the target object based on the distance to each of the aforementioned measurement points, A step of estimating the volume of the object based on the three-dimensional shape of the object, The steps include receiving input for the required mass, which is the required mass of the object in question, A step of calculating the required volume corresponding to the received required mass based on the density corresponding to the type of object, Based on the estimated volume of the target object, the step of calculating a region in the three-dimensional shape of the target object corresponding to the calculated required volume, The steps include: displaying a reference marker image indicating the calculated area on a display provided in the display device so as to superimpose it on the target object; A display control method including the following.
2. A display control method according to claim 1, The captured image is an image of the external scenery of the display device captured by the camera. Display control method.
3. A display control method according to claim 1, The display device is a head-mounted display or smart glasses. Display control method.
4. A display control method according to any one of claims 1 to 3, The aforementioned processor, A step of calculating the mass of the target object based on the density corresponding to the type of target object and the estimated volume of the target object, The steps include displaying an image on the display showing the mass of the object, A display control method including the following.
5. A display control method according to any one of claims 1 to 3, The aforementioned processor, A step of accepting inputs for multiple required masses, The steps include displaying a plurality of reference marker images, each representing one of the plurality of required masses, on the display so as to superimpose them onto the target object, A display control method including the following.
6. A display control method according to claim 5, The processor further displays an augmented reality image on the display showing weighing result fields indicating the plurality of required masses. A display control method including the following.
7. A display control method according to any one of claims 1 to 3, When the processor receives an input for a display position switching instruction, it takes the step of displaying an image on the display that shows a new reference marker different from the reference marker image, superimposed on the target object. A display control method including the following.
8. A display control method according to any one of claims 1 to 3, If the processor cannot recognize the three-dimensional shape of the target object based on the distance to each measurement point, it provides a notification to the user of the display device prompting them to take an image of the target object from a different angle. A display control method including the following.
9. A display control method according to any one of claims 1 to 3, The processor determines whether the object in question is a solid, a container, or a liquid, and according to the result of this determination, selects and displays one of the following as the reference marker image: a display along the outer surface of the object, a display along the containment space of the container, or a display where the liquid surface is horizontal and the contour follows the inner surface of the container. A display control method including the following.