Mixed reality device, processing method, program, and storage medium

Abstract

The invention relates to a mixed reality device, a processing method, a program, and a storage medium. Provided is a mixed reality device having better convenience. Also provided are a processing method, a program, and a storage medium that make it possible to further improve the convenience of a mixed reality device. A mixed reality device according to an embodiment sets a base point of a virtual space using a mark provided in a real space. The mixed reality device displays a virtual object at a preset position in a three-dimensional coordinate system based on the base point. The mixed reality device acquires a movement amount of the mark with respect to a reference position of the mark when the base point is set. When the amount of movement exceeds a preset threshold value, the mixed reality device moves the base point in accordance with the amount of movement.

Description

Technical Field

[0001] Embodiments of the present invention relate to a mixed reality device, a processing method, a program, and a storage medium. Background Art

[0002] In the past, mixed reality devices were used to perform work efficiently. Mixed reality devices can display virtual space and real space in an overlapping manner and provide various information to operators. Operators can perform work more efficiently by referring to the information displayed by the mixed reality device. Mixed reality devices are required to have technology that can further improve convenience.

[0003] Patent Document 1: International Publication No. 2021 / 176645 Summary of the invention

[0004] The problem to be solved by the present invention is to provide a mixed reality device with better convenience. In addition, the problem to be solved by the present invention is to provide a processing method, a program, and a storage medium that can further improve the convenience of the mixed reality device.

[0005] The mixed reality device of the embodiment uses a marker set in the real space to set a base point of the virtual space. The mixed reality device displays a virtual object at a preset position in a three-dimensional coordinate system based on the base point. The mixed reality device obtains the movement amount of the marker relative to the reference position of the marker when the base point is set. When the movement amount exceeds a preset threshold, the mixed reality device moves the base point according to the movement amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic diagram of a mixed reality device of an exemplary embodiment.

[0007] Figure 2 It is a schematic diagram which illustrates the object of the work.

[0008] Figure 3 is a schematic diagram showing an output example of a mixed reality device according to an implementation manner.

[0009] Figure 4 is a schematic diagram showing an output example of a mixed reality device according to an implementation manner.

[0010] Figure 5 It is a schematic diagram illustrating a working situation.

[0011] Figure 6 This is a schematic diagram showing an example of a tool.

[0012] Figure 7 It is a schematic diagram illustrating a working situation.

[0013] Figure 8 (a) and Figure 8 (b) is a schematic diagram for illustrating the processing performed by the mixed reality device of the implementation mode.

[0014] Fig. 9 This is a schematic diagram for explaining the calculation method of the movement amount.

[0015] Fig.10 This is a schematic diagram for explaining the calculation method of the movement amount.

[0016] Fig.11 It is a flowchart showing the processing method of the implementation mode.

[0017] Fig.12 This is a flowchart showing a method for correcting a display position of a virtual object.

[0018] Fig.13 This is a flowchart showing a method for controlling the operation of the MR system.

[0019] Fig.14 This is a schematic diagram showing a display example of a mixed reality device of a modified example.

[0020] Fig.15 This is a schematic diagram showing the hardware configuration.

[0021] Explanation of symbols

[0022] 1: Photographic device; 2: Processing device; 90: Computer; 91: CPU; 92: ROM; 93: RAM; 94: Storage device; 95: Input interface; 96: Output interface; 97: Communication interface; 98: System bus; 100: MR device; 101: Frame; 111, 112: Lens; 121, 122: Projection device; 131: Image camera; 132: Depth camera; 140: Sensor; 141: Microphone; 150: Processing device; 160: Battery; 170: Storage device; 170a: Job master data; 170b : Base point master data; 170c: Tool master data; 170d: Fastening part master data; 170e: History data; 200: Item; 201~208: Fastening part; 210, 215: Mark; 221: Trolley; 2: Caster; 223: Sensor; 251: Left hand; 252: Right hand; 261, 262: Virtual object; 280: Wrench; 281: Mark; 282: Head; 290: Extension rod; 301~308, 311~318: Virtual object; 320: Message; M: Processing method; M1: Correction method; M2: Control method. DETAILED DESCRIPTION

[0023] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. The accompanying drawings are schematic or conceptual drawings, and the relationship between the thickness and width of each part, the ratio of the size between the parts, etc. may not be the same as the actual situation. In addition, even when representing the same part, the size and ratio of each other are sometimes represented differently according to the accompanying drawings. In the specification of this application and each figure, the same symbols are marked for the same elements as those already described, and the detailed description is appropriately omitted.

[0024] Figure 1 is a schematic diagram of a mixed reality device in accordance with an exemplary embodiment.

[0025] Embodiments of the present invention relate to mixed reality devices (MR devices). Figure 1 As shown, the MR device 100 of the embodiment includes a frame 101 , a lens 110 , a lens 112 , a projection device 121 , a projection device 122 , an image camera 131 , a depth camera 132 , a sensor 140 , a microphone 141 , a processing device 150 , a battery 160 , and a storage device 170 .

[0026] In the example shown in the figure, the MR device 100 is a binocular head mounted display. Two lenses 111 and 112 are embedded in a frame 101. The projection device 121 and the projection device 122 project information onto the lens 111 and the lens 112, respectively.

[0027] Projection devices 121 and 122 display the recognition result of the operator's body, virtual objects, etc. on lenses 111 and 112. Only one of projection devices 121 and 122 may be provided, and information may be displayed on only one of lenses 111 and 112.

[0028] The lenses 111 and 112 are light-transmissive. The wearer of the MR device 100 can visually check the situation in the real space through the lenses 111 and 112. In addition, the wearer of the MR device 100 can also visually check the information projected onto the lenses 111 and 112 by the projection devices 121 and 122. The information (virtual space) is displayed overlapping the real space through the projection of the projection devices 121 and 122.

[0029] The image camera 131 detects visible light and obtains a two-dimensional image. The depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. The sensor 140 is a six-axis detection sensor capable of detecting three-axis angular velocity and three-axis acceleration. The microphone 141 receives sound input.

[0030] The processing device 150 controls each element of the MR device 100. For example, the processing device 150 controls the display of the projection device 121 and the projection device 122. The processing device 150 detects the movement of the field of view based on the detection result of the sensor 140. The processing device 150 changes the display of the projection device 121 and the projection device 122 according to the movement of the field of view. In addition, the processing device 150 can perform various processes using data obtained from the image camera 131 and the depth camera 132, data from the storage device 170, and the like.

[0031] The battery 160 supplies power required for operation to each element of the MR device 100. The storage device 170 stores data required for processing by the processing device 150, data obtained by processing by the processing device 150, and the like. The storage device 170 may be provided outside the MR device 100 and communicate with the processing device 150.

[0032] The MR device of the embodiment is not limited to the example shown in the figure, and may be a single-eye head mounted display. The MR device may be a glasses type as shown in the figure, or a helmet type.

[0033] Figure 2 It is a schematic diagram which illustrates the object of the work.

[0034] When the operator wears the MR device 100, for example, Figure 2 The article 200 shown is subjected to work. The article 200 is a cylindrical component having fastening parts 201 to 208. The worker uses a wrench and an extension rod to tighten the screws at the fastening parts 201 to 208 one by one.

[0035] When performing work using the MR device 100, a three-dimensional coordinate system of a virtual space is set based on a predetermined object. Figure 2 In the example shown, a marker 210 is provided near the work object. The marker 210 is an AR marker. As described later, the marker 210 is provided to set a base point of a three-dimensional coordinate system. Instead of an AR marker, a one-dimensional code (barcode), a two-dimensional code (QR code (registered trademark)), etc. may be used as the marker 210.

[0036] When the operation starts, the image camera 131 and the depth camera 132 take a picture of the marker 210. The processing device 150 recognizes the marker 210 based on the photographed image. The processing device 150 sets the base point of the virtual space based on the position and orientation of the marker 210. When the base point is set, the three-dimensional coordinate system of the virtual space is determined. By setting the base point based on a predetermined object existing in the real space, the virtual object can be displayed corresponding to the object in the real space.

[0037] The processing device 150 also calculates the position of the MR device 100. As an example, the processing device 150 calculates the position and direction of the MR device 100 through the function of spatial mapping. In the MR device 100, the distance between the MR device 100 and the surrounding objects is measured by the depth camera 132. Based on the measurement result of the depth camera 132, the surface information of the surrounding objects is obtained. The surface information includes the position and direction of the surface of the object. For example, the surface of each object is represented by a plurality of grids, and the position and direction are calculated for each grid. Based on the surface information, the processing device 150 calculates the relative position and direction of the MR device 100 relative to the surface of the surrounding objects. When the marker 210 is recognized, the position of each surface is also represented by a three-dimensional coordinate system with the marker 210 as the base point. Based on the positional relationship between the surface of the object and the MR device 100, the position and direction of the MR device 100 in the three-dimensional coordinate system are calculated. Spatial mapping is repeatedly performed at a specified interval.

[0038] Figure 3 is a schematic diagram showing an output example of a mixed reality device according to an implementation manner.

[0039] During the operation, the object 200, the left hand 251 of the operator, and the right hand 252 of the operator are photographed by the image camera 131 and the depth camera 132. The processing device 150 recognizes the left hand 251 and the right hand 252 from the photographed images by hand tracking. The processing device 150 can also display the recognition result on the lens 111 and the lens 112 through the projection device 121 and the projection device 122. Hereinafter, the case where the processing device uses the projection device to display information on the lens is also referred to as "the processing device displays information".

[0040] For example, Figure 3 As shown, the processing device 150 displays the recognition results of the left hand 251 and the recognition results of the right hand 252 overlapping with the hands in the real space. In the illustrated example, multiple virtual objects 261 and multiple virtual objects 262 are displayed as the recognition results of the left hand 251 and the right hand 252. The multiple virtual objects 261 respectively represent multiple joints of the left hand 251. The multiple virtual objects 262 respectively represent multiple joints of the right hand 252. Instead of the joints, virtual objects (grids) representing the surface shapes of the left hand 251 and the right hand 252 may be displayed. The grid is obtained by spatial mapping.

[0041] The processing device 150 measures the position of each hand when the left hand 251 and the right hand 252 are identified. Specifically, the hand includes multiple joints such as the DIP joint, the PIP joint, the MP joint, and the CM joint. The position of any one of these joints can be used as the position of the hand. The center of gravity of multiple joints can also be used as the position of the hand. Alternatively, the center position of the entire hand can also be used as the position of the hand.

[0042] Figure 4 is a schematic diagram showing an output example of a mixed reality device according to an implementation manner.

[0043] like Figure 4 As shown, the processing device 150 displays virtual objects 301 to 308 and virtual objects 311 to 318. The virtual objects 311 to 318 are displayed at positions away from the fastening parts 201 to 205. The virtual objects 311 to 318 are displayed between the fastening parts 201 to 205 and the virtual objects 301 to 308. The virtual objects 311 to 318 indicate which fastening parts the virtual objects 301 to 308 correspond to.

[0044] In the example shown in the figure, virtual objects 301 to 308 are spherical, and virtual objects 311 to 318 are rod-shaped. As long as the operator can visually confirm each virtual object, the shape of each virtual object is not limited to this example. For example, virtual objects 301 to 308 may be cubes, and virtual objects 311 to 318 may be linear. Figure 4 The fastening locations 206 to 208 not shown in the figure also display virtual objects that are the same as the virtual objects 301 to 308 and the virtual objects 311 to 318 .

[0045] Figure 5 It is a schematic diagram illustrating a working situation.

[0046] For example, Figure 5 As shown, a wrench 280 and an extension rod 290 are used to tighten a screw at a fastening portion. As an example, when tightening a screw at a fastening portion 203, an operator places the screw in a threaded hole of the fastening portion 203. The operator fits one end of the extension rod 290 into the screw. The operator fits the head of the wrench 280 into the other end of the extension rod 290. The operator may also press the head of the wrench 280 with one hand and hold the handle of the wrench 280 with the other hand. By turning the wrench 280, the screw is tightened at the fastening portion 203 via the extension rod 290.

[0047] At this time, the operator places the extension rod 290 in such a manner that the extension rod 290 is close to or in contact with the virtual object 313. In addition, the operator holds the head of the wrench 280 in such a manner that the hand is in contact with the virtual object 303. By displaying the virtual object, the operator can easily grasp the position of the tool and the hand when tightening the screw relative to the fastening part 203. Thus, the working efficiency can be improved.

[0048] After displaying the virtual object, the processing device 150 may also determine whether a specified object is in contact with the virtual object. For example, the processing device 150 determines whether the hand is in contact with the virtual object 303. Specifically, the processing device 150 calculates the distance between the position of the hand and the virtual object 303. When the distance is less than a preset threshold, the processing device 150 determines that the hand is in contact with the virtual object. Figure 5 In FIG. 1 , the diameter of the virtual object 303 (sphere) corresponds to the threshold value. The sphere indicates the range within which the hand is judged to be in contact with the virtual object.

[0049] Figure 6 This is a schematic diagram showing an example of a tool.

[0050] The processing device 150 can also determine whether the tool is in contact with the virtual object 303. For example, Figure 6 As shown, a plurality of markers 281 are mounted on the wrench 280. The processing device 150 recognizes the plurality of markers 281 based on the image captured by the image camera 131. The processing device 150 measures the position of each marker 281. The positional relationship between the plurality of markers 281 and the head 282 of the wrench 280 is pre-registered. The processing device 150 calculates the position of the head 282 based on the positions of at least three recognized markers 281 and the pre-registered positional relationship. The processing device 150 calculates the distance between the position of the head 282 and the virtual object 303. When any of the distances is less than a preset threshold value, the processing device 150 determines that the wrench 280 is in contact with the virtual object.

[0051] When the distance is smaller than the threshold value, it can be inferred that the screw is being tightened with respect to the fastening portion corresponding to the virtual object 303. Figure 5 In the example shown, based on the contact between the hand and the virtual object 303 , it is estimated that work is being performed on the fastening location 203 corresponding to the virtual object 303 .

[0052] Figure 7 It is a schematic diagram illustrating a working situation. Figure 8 (a) and Figure 8 (b) is a schematic diagram for illustrating the processing performed by the mixed reality device of the implementation mode.

[0053] For example, Figure 7As shown, the article 200 is placed on a trolley 221. Casters 2 are mounted on the lower surface of the trolley 221. The trolley 221 is movable relative to the ground. The marker 210 is mounted on the trolley 221, for example. The article 200 is placed so as not to move relative to the trolley 221 during operation. Therefore, the positional relationship between the article 200 and the marker 210 is substantially fixed.

[0054] The virtual object is displayed at a preset position in the three-dimensional coordinate system based on the marker 210. Figure 5 In the example shown, the position of the fastening portion 203, the length of the extension rod 290, etc. are registered in advance. The virtual object 313 is displayed between the position of the fastening portion 203 and the position separated from the fastening portion 203 by the length of the extension rod 290. The virtual object 303 is displayed at the position separated from the fastening portion 203 by the length of the extension rod 290. As a result, the virtual objects 303 and 313 are displayed corresponding to the fastening portion 203.

[0055] The operator can move the object 200 to facilitate the work. For example, the operator rotates the object 200 during the work so that the fastening part of the screw is located near the operator. At this time, if the position of the base point is not updated and the object 200 is moved, the display position of the virtual object deviates from the position where it should be displayed. As a result, Figure 8 As shown in FIG. 2( a ), the virtual objects 303 and 313 are displayed offset from the fastening portion 203 .

[0056] The processing device 150 obtains the movement amount of the object 200 in the display of the virtual object so as to appropriately display the virtual object. Here, "movement" includes translational movement and rotational movement. "Movement amount" includes the amount of translational movement and the amount of rotational movement. When the movement amount exceeds a preset threshold, the processing device 150 moves the base point according to the movement amount. As a result, Figure 8 As shown in (b) of FIG. 1 , the display positions of the virtual objects 303 and 313 are changed. The virtual objects 303 and 313 are displayed at more appropriate positions.

[0057] Fig. 9 as well as Fig.10 This is a schematic diagram for explaining the calculation method of the movement amount.

[0058] The movement amount is calculated using any of the following first to fourth methods.

[0059] In the first method, Fig. 9As shown, a photographing device 1 is set at a place where an object 200 is set. The photographing device 1 photographs the object 200. A mark 215 is installed at a position of the object 200 that can be observed from the photographing device 1. The mark 215 is different from the mark 210. As long as the position of the mark 215 is fixed relative to the object 200, the mark 215 can also be installed on an object other than the object 200. In order to improve the recognition accuracy of the mark 215, the mark 215 preferably has a color that is easily distinguished from other objects. In addition, in order to improve the calculation accuracy of the movement amount, it is preferred to install a plurality of marks 215.

[0060] The photographing device 1 photographs the marker 215 and obtains an image. The processing device 2 recognizes the marker 215 from the image and calculates the position of the marker 215 in the image. The processing device 2 holds the position of the marker 215 at the time when the three-dimensional coordinate system is set by the processing device 150 as the reference position. When the processing device 2 obtains the image, it calculates the difference between the reference position and the position of the recognized marker 215 as the movement amount.

[0061] In the second method, similar to the first method, the photographing device 1 photographs the object 200. The marker 215 is not required. The processing device 2 recognizes the object 200 from the image and calculates the position of the object 200 in the image. A pre-learned model is used for the recognition of the object 200. In order to improve the accuracy of recognition, the model preferably includes a convolutional neural network (CNN). The processing device 2 maintains the position of the object 200 whose three-dimensional coordinate system is set by the processing device 150 as a reference position. When the processing device 2 obtains the image, it calculates the difference between the reference position and the position of the recognized object 200 as the movement amount.

[0062] In the third method, a sensor 223 for obtaining the amount of movement is used. The sensor 223 is mounted on the object 200 or the trolley 221. As long as the position of the sensor 223 is fixed relative to the object 200, the sensor 223 may be mounted on an object other than the object 200 or the trolley 221. For example, the sensor 223 includes one or more selected from an acceleration sensor or an angular velocity sensor. The processing device 2 calculates the amount of movement of the object 200 based on the acceleration or angular velocity detected by the sensor 223. Alternatively, the sensor 223 may also include a radio wave receiver, and an indoor positioning system is used to obtain the amount of movement of the object 200. In indoor positioning, beacons, radio frequency identification (RFID), indoor information system (IMES), ultra-wideband (UWB) wireless communications, etc. are used.

[0063] In the fourth method, the spatial mapping function of the MR device 100 is used. In the spatial mapping, the surface information of the surrounding objects is calculated based on the depth image obtained by the depth camera 132. The surface position change is calculated by repeatedly calculating the surface information. In addition, the movement amount of the MR device 100 is calculated based on the detection result of the sensor 140. The difference between the surface position change of the object 200 and the movement amount of the MR device 100 is equivalent to the actual movement amount of the object 200.

[0064] When the movement amount is calculated by any one of the first to fourth methods, the processing device 150 obtains the movement amount. The processing device 150 compares the movement amount with a threshold. The threshold is pre-set based on the size of the virtual object, the allowable offset of the display position, etc. The movement amount is repeatedly calculated at a predetermined interval. When the object 200 moves, in order to shorten the period during which the virtual object is offset relative to the object 200, it is preferable that the calculation interval is short.

[0065] Fig.11 It is a flowchart showing the processing method of the implementation mode.

[0066] In execution Fig.11 Prior to the processing method M shown, the work master data 170 a , the base point master data 170 b , the tool master data 170 c , and the fastening location master data 170 d are prepared in advance. Each of the master data is stored in the storage device 170 .

[0067] First, select the job to be performed (step S1). The job ID, job name, item ID, and item name are registered in the job master data 170a. The job is determined by the job ID, job name, ID of the item to be performed, or the name of the item. The processing device 150 accepts the selection of the job. For example, the operator selects the job to be performed. The job to be performed may also be selected by the upper system, and the processing device 150 accepts the selection. The processing device 150 may also determine the next job to be performed based on the data obtained from the image camera 131 or other sensors. The processing device 150 selects the job based on the determination result.

[0068] Next, the image camera 131 photographs the mark 210. The processing device 150 sets the base point of the three-dimensional coordinate system based on the position and orientation of the mark 210 (step S2). At this time, the processing device 150 refers to the base point master data 170b. The base point master data 170b stores the method for setting the base point for each operation. The processing device 150 obtains the method for setting the base point in the selected operation and sets the base point according to the method.

[0069] After the base point is set, the processing device 150 displays the virtual object (step S3). When the virtual object is displayed, the data of the tool master data 170c and the fastening part master data 170d are appropriately referred to.

[0070] In the tool master data 170c, the ID of the tool used, the model of the tool, the length of the tool, the model of the sleeve, the length of the sleeve, etc. are stored for each operation. The model of the tool indicates the classification of the tool based on the structure, appearance or performance. The length of the tool is the length from the rotation center to the handle when the tool is used in threaded fastening. The model of the sleeve indicates the classification of the sleeve based on the structure or appearance. The length of the sleeve indicates the length of the sleeve in the direction in which the tool is connected to the screw when the screw is tightened. The processing device 150 obtains the data of the tool used in the operation selected in step S1 from the tool master data 170c. When an extension rod is used, the model and length of the extension rod are also stored in the tool master data 170c. The processing device 150 further obtains data related to the extension rod from the tool master data 170c.

[0071] In the fastening part master data 170d, the fastening part ID, the position of the fastening part, the required torque value, and the number of screw tightening times are stored for each fastening part. The fastening position indicates the position where the fastening part exists, and is determined by the position of the three-dimensional coordinate system set in step S2. The number of screw tightening times indicates the number of screw tightening times required for each fastening part. When a mark is added to the screw after tightening, the color of the mark is also registered.

[0072] The processing device 150 identifies a predetermined object from the image captured by the image camera 131 (step S4). The "predetermined object" is an object determined to be in contact with the virtual object. The processing device 150 calculates the position of the predetermined object and determines whether the distance between the object and the virtual object is less than a threshold value (step S5). For example, when the distance is less than the threshold value, the predetermined object is in contact with the virtual object.

[0073] When the distance is greater than the threshold, step S4 is executed again. When the distance is less than the threshold, the processing device 150 infers that work is being performed on the part corresponding to the virtual object.

[0074] The processing device 150 records the operation related to the inferred part and saves it in the history data 170e (step S6). For example, the torque value detected by the tool is associated with the ID of the operation and the ID of the inferred part. As shown in the figure, the processing device 150 can also further associate the model and ID of the tool used, the number of screw tightenings, and the recognition result of the mark with the ID of the tightening part. The mark is recognized from the image photographed by the image camera 131 by the processing device 150. The processing device 150 extracts a block of pixels of the mark color from the image, and the number of pixels of the block is counted. When the number of pixels exceeds a preset threshold, it is determined that a mark is attached.

[0075] The processing device 150 determines whether the operation is completed (step S7). If the operation is not completed, step S3 is executed again.

[0076] Fig.12 This is a flowchart showing a method for correcting a display position of a virtual object.

[0077] Fig.12 The correction method M1 shown in Fig.11 The calibration method M1 is repeated until step S6 is executed.

[0078] First, the processing device 150 obtains the movement amount of the object on which the virtual object is displayed (step S11). The movement amount is calculated by one or more of the above-mentioned methods 1 to 4. The processing device 150 determines whether the movement amount exceeds a threshold value (step S12). If the movement amount exceeds the threshold value, the processing device 150 moves the base point according to the movement amount (step S13). For example, the base point is translated by the same amount as the translational movement of the object 200, and the base point is rotated by the same amount as the rotational movement of the object 200. The processing device 150 redisplays the virtual object based on the moved base point (step S14). If the movement amount does not exceed the threshold value in step S12 or after step S14, the processing device 150 determines whether the execution is performed. Fig.11 Step S6 (step S15) is shown. If step S6 is not executed, step S11 is executed again.

[0079] In addition, when the marker 210 is photographed after the base point is moved, the base point is reset using the marker 210. The display position of the virtual object is set based on the base point using the marker 210. Therefore, by resetting the base point using the marker 210, the virtual object can be displayed at the most preferred position.

[0080] Advantages of the embodiment will be described.

[0081] MR equipment is effective in improving work efficiency. For example, Figure 5 As shown, by displaying the virtual object, the operator can easily understand which fastening part is better to tighten the screw. In addition, when tightening the screw relative to the fastening part, the operator can intuitively understand the position of the configuration tool and the hand. As a result, the work efficiency can be improved. In addition, it is also possible to make a work record based on the proximity or contact between a specified object and a virtual object. By automatically making the work record, there is no need for the operator to make the work record. It can avoid making an erroneous work record due to the operator's mistake.

[0082] These advantages can be obtained by displaying a virtual object at a specific position corresponding to an object in real space. The display position of the virtual object in the three-dimensional coordinate system is preset. The base point of the three-dimensional coordinate system is set using a marker set in real space. Therefore, when the positional relationship between the marker recognized by the MR device 100 and the object in real space changes, the display position of the virtual object is offset relative to the object in real space. As a result, the operator is confused or a work record is made for a part different from the actual work part.

[0083] As an example, from Figure 7 From the state shown in FIG. 1 , the operator rotates the object 200 and the mark 210, but does not photograph (recognize) the rotated mark 210. In this case, the positional relationship between the identified mark 210 and the object 200 changes due to the rotation. As a result, Figure 8 As shown in (a) of FIG. 1 , the virtual objects 303 and 313 are displayed offset from the fastening location 203 .

[0084] Regarding this problem, in an embodiment of the present invention, the processing device 150 moves the base point according to the movement amount. By moving the base point, it is possible to prevent the virtual object from being displayed offset from the desired position. As a result, Figure 8 As shown in (b), even if the object 200 moves, the display positions of the virtual objects 303 and 313 can be moved according to the movement of the object 200. By displaying the virtual objects at appropriate positions, it is easier for the operator to perform the operation. In addition, the operation record can be made more appropriately. According to the embodiment of the present invention, the convenience of the MR device 100 can be improved.

[0085] When calculating the amount of movement, two or more of the first to fourth methods may be used. For example, the first method and any one of the third or fourth methods are executed respectively. Regarding the first method, the amount of calculation required is small and the accuracy is also good. Therefore, the amount of movement calculated by the first method takes precedence over the amount of movement calculated by other methods. On the other hand, when the mark 215 is blocked by other objects, the calculated amount cannot be obtained by the first method. In this case, the amount of movement calculated by the third method or the fourth method is used.

[0086] In addition, in order to suppress the offset between the virtual object and the object, there is also a method of identifying the object and its details (fastening parts, etc.) from the image. In this case, the virtual object can be displayed in accordance with the identified details. Therefore, even if the operator moves the object, the moved object can be identified and the virtual object can be displayed. However, in order to accurately identify the object and its details from the image, a large amount of calculation is required. It takes a longer time from the time the operator moves the object to the time the virtual object is displayed at the desired position.

[0087] In an embodiment, a marker is used in the setting of a base point, and a virtual object is displayed at a specified position. According to this method, the required amount of calculation can be suppressed. In addition, by using an image of the marker 215 or a detection result of the sensor 223 to calculate the amount of movement, the required amount of calculation can be further suppressed. According to an embodiment of the present invention, the amount of calculation can be reduced, and the display position of the virtual object can be updated at a shorter interval. For example, the display position of the virtual object can be updated in real time according to the movement of the object.

[0088] (Variation Example)

[0089] Fig.13 This is a flowchart showing a method for controlling the operation of the MR system. Fig.14 This is a schematic diagram showing a display example of a mixed reality device of a modified example.

[0090] Can also replace Fig.12 The calibration method M1 shown is performed Fig.13 In the control method M2, in steps S21 and S22, the movement amount is obtained in the same manner as in steps S11 and S12 of the correction method M1, and the movement amount is compared with a threshold value. If the movement amount exceeds the threshold value, the processing device 150 stops. Fig.11 The processing method M shown in FIG. 2 is used for processing (step S23). As a result, it becomes impossible to determine the contact between the predetermined object and the virtual object. The processing device 150 may also stop displaying the virtual object. As a result, the creation of the operation record is also stopped.

[0091] The processing device 150 outputs an alarm (step S24). Fig.14 As shown, a message 320 is displayed to the operator so that the display position of the virtual object is re-determined. When the message 320 is displayed, the operator faces the mark 210. The mark 210 is photographed by the image camera 131, and the base point is reset based on the mark 210. In addition to the message 320, sound, vibration, light, etc. can also be output as an alarm.

[0092] The processing device 150 determines whether the base point has been reset (step S25). When the base point has been reset, the processing device 150 redisplays the virtual object based on the reset base point (step S26). The processing device 150 restarts the processing method M (step S27). Thus, it is possible to determine the contact between the specified object and the virtual object. As a result, the operation record is also automatically created.

[0093] If the movement amount does not exceed the threshold value in step S22 or after step S27, the processing device 150 determines whether the Fig.11 Step S6 (step S28) is shown. If step S6 is not executed, step S21 is executed again.

[0094] According to a variation of the embodiment, when the amount of movement exceeds a threshold, the Fig.11 The processing method M shown in the figure is used. As a result, the contact between the predetermined object and the virtual object is not determined, and the estimation of the fastening part being operated is also stopped. The creation of the operation record is also stopped. As a result, it is possible to prevent the erroneous estimation that other fastening parts are being operated.

[0095] According to the modification, it is possible to avoid making an erroneous work record due to the movement of the article 200. Therefore, the convenience of the MR device 100 can be improved.

[0096] When the processing is stopped, the processing device 150 preferably outputs an alarm to prompt the operator to reset the base point. By resetting the base point earlier, the work can be restarted earlier. Therefore, the reduction in work efficiency can be suppressed.

[0097] Here, the example in which the embodiment of the present invention is applied to the operation of tightening a screw is mainly described. The embodiment of the present invention can also be applied to the operation of loosening a screw. Even in the case of loosening a screw, Figure 5 As shown in FIG. 1 , a tool is used to rotate a screw. In this case, by displaying a virtual object, the work can be performed efficiently.

[0098] Embodiments of the present invention can also be applied to operations other than the operation of rotating a screw. For example, the operation can also be the assembly of an object, the disassembly of an object, or the transportation of an object. In any operation, the operation can be effectively performed by displaying a virtual object. In addition, when the object of the operation moves, the display position of the virtual object is changed according to the amount of movement, or the inference of the part being operated is stopped, thereby improving the convenience of the MR device.

[0099] Fig.15 This is a schematic diagram showing the hardware configuration.

[0100] As the processing device 2 and the processing device 150, for example, Fig.15 The computer 90 shown in FIG. 9 includes a CPU 91 , a ROM 92 , a RAM 93 , a storage device 94 , an input interface 95 , an output interface 96 , and a communication interface 97 .

[0101] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for the computer 90 to implement the above-mentioned processing. The RAM 93 functions as a storage area for developing the program stored in the ROM 92.

[0102] The CPU 91 includes a processing circuit. The CPU 91 uses the RAM 93 as a working memory and executes a program stored in at least one of the ROM 92 and the storage device 94. While executing the program, the CPU 91 controls each component via the system bus 98 and executes various processes.

[0103] The storage device 94 stores data necessary for executing the program and data obtained by executing the program. The storage device 94 includes a solid state drive (SSD) or the like.

[0104] The input interface (I / F) 95 can connect the computer 90 to an input device. The CPU 91 can read various data from the input device via the input I / F 95.

[0105] The output interface (I / F) 96 can connect the computer 90 to an output device. The CPU 91 can send data to the output device via the output I / F 96 and cause the output device to output information.

[0106] The communication interface (I / F) 97 can connect a device external to the computer 90 to the computer 90. The communication I / F 97 connects a digital tool to the computer 90 through, for example, Bluetooth (registered trademark) communication.

[0107] The data processing performed by the processing device 2 and the processing device 150 can be performed by only one computer 90. A part of the data processing can also be performed by a server or the like via the communication I / F 97. The processing of the processing device 2 can also be performed by the processing device 150. In the above example, a part of the processing performed by the processing device 150 can also be performed by the processing device 2.

[0108] The processing of the above-mentioned various data can also be recorded as a program that can be executed by a computer, on a disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0109] For example, the information recorded in the recording medium can be read out by a computer (or embedded system). In the recording medium, the recording format (storage format) is arbitrary. For example, the computer reads out the program from the recording medium, and the CPU executes the instructions recorded in the program based on the program. In the computer, the acquisition (or reading) of the program can also be carried out through a network.

[0110] The embodiments of the present invention include the following features.

[0111] (Feature 1)

[0112] A mixed reality device, wherein

[0113] Use the markers set in the real space to set the base point of the virtual space.

[0114] In the three-dimensional coordinate system based on the base point, a virtual object is displayed at a preset position.

[0115] obtaining the movement amount of the mark relative to the reference position of the mark when the reference point is set,

[0116] When the movement amount exceeds a preset threshold value, the base point is moved according to the movement amount.

[0117] (Feature 2)

[0118] The mixed reality device according to feature 1, wherein:

[0119] recognizing the mark from an image obtained by photographing the mark,

[0120] The above base point is set based on the position and orientation of the above mark.

[0121] (Feature 3)

[0122] The mixed reality device according to feature 2, wherein:

[0123] When the marker is re-photographed after the base point has been moved, the base point is reset based on the position and orientation of the marker recognized from the re-photographed image.

[0124] (Feature 4)

[0125] The mixed reality device according to any one of features 1 to 3, wherein:

[0126] Repeatedly obtain the above movement amount,

[0127] Whenever the movement amount is obtained, it is determined whether the movement amount exceeds the threshold value.

[0128] (Feature 5)

[0129] The mixed reality device according to any one of features 1 to 4, wherein:

[0130] The movement amount is calculated using an image of the object being worked on or a detection result from a sensor fixed relative to the object.

[0131] (Feature 6)

[0132] The mixed reality device according to any one of features 1 to 5, wherein:

[0133] The virtual object is displayed corresponding to the fastening location of the object being worked on.

[0134] (Feature 7)

[0135] The mixed reality device according to feature 6, wherein:

[0136] Calculate the distance between the specified object and the virtual object.

[0137] When the distance is smaller than a preset threshold value, it is inferred that work is being performed on the fastened portion.

[0138] (Feature 8)

[0139] A mixed reality device, wherein

[0140] Use the markers set in the real space to set the base point of the virtual space.

[0141] Display virtual objects corresponding to the fastening parts of the items.

[0142] When the distance between the predetermined object and the virtual object is smaller than a preset threshold, it is inferred that work is being performed on the fastening portion.

[0143] When the amount of movement of the mark relative to the reference position of the mark when the base point is set exceeds a preset threshold value, the estimation of the fastening location being operated is stopped.

[0144] (Feature 9)

[0145] The mixed reality device according to feature 8, wherein:

[0146] After the estimation is stopped, when the base point is reset, the estimation of the fastening location being operated is restarted.

[0147] (Feature 10)

[0148] The mixed reality device according to feature 8 or 9, wherein:

[0149] Repeatedly obtain the above movement amount,

[0150] Whenever the movement amount is obtained, it is determined whether the movement amount exceeds the threshold value.

[0151] (Feature 11)

[0152] A processing method, wherein

[0153] Mixed reality devices are,

[0154] Use the markers set in the real space to set the base point of the virtual space.

[0155] In the three-dimensional coordinate system based on the base point, a virtual object is displayed at a preset position.

[0156] obtaining the movement amount of the mark relative to the reference position of the mark when the reference point is set,

[0157] When the movement amount exceeds a preset threshold value, the base point is moved according to the movement amount.

[0158] (Feature 12)

[0159] A procedure in which

[0160] Enable the mixed reality device to execute the processing method recorded in feature 11.

[0161] (Feature 13)

[0162] A storage medium, wherein

[0163] The program described in feature 12 is stored.

[0164] According to the above-described embodiments, a mixed reality device with better convenience is provided. In addition, a processing method, a program, and a storage medium that can further improve the convenience of the mixed reality device are provided.

[0165] In the present specification, "or" means that "at least one or more" of the items listed in the article can be adopted.

[0166] Several embodiments of the present invention have been illustrated above, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments and their variations are included in the scope and main purpose of the invention, and are included in the invention described in the scope of the patent claim and the scope equivalent thereto. In addition, the above-mentioned embodiments can be implemented in combination with each other.

Claims

1. A mixed reality device, wherein: Use the markers set in the real space to set the base point of the virtual space. In the three-dimensional coordinate system based on the base point, a virtual object is displayed at a preset position. obtaining the movement amount of the mark relative to the reference position of the mark when the reference point is set, When the movement amount exceeds a preset threshold value, the base point is moved according to the movement amount.

2. The mixed reality device according to claim 1, wherein: recognizing the mark from an image obtained by photographing the mark, The above base point is set based on the position and orientation of the above mark.

3. The mixed reality device according to claim 2, wherein: When the marker is re-photographed after the base point has been moved, the base point is reset based on the position and orientation of the marker recognized from the re-photographed image.

4. The mixed reality device according to claim 1, wherein: Repeatedly obtain the above movement amount, Whenever the movement amount is obtained, it is determined whether the movement amount exceeds the threshold value.

5. The mixed reality device according to claim 1, wherein: The movement amount is calculated using an image of the object being worked on or a detection result from a sensor fixed relative to the object.

6. The mixed reality device according to any one of claims 1 to 5, wherein: The virtual object is displayed corresponding to the fastening location of the object being worked on.

7. The mixed reality device according to claim 6, wherein: Calculate the distance between the specified object and the virtual object. When the distance is smaller than a preset threshold value, it is inferred that work is being performed on the fastened portion.

8. A mixed reality device, wherein: Use the markers set in the real space to set the base point of the virtual space. Display virtual objects corresponding to the fastening parts of the items. When the distance between the predetermined object and the virtual object is smaller than a preset threshold, it is inferred that work is being performed on the fastening portion. When the amount of movement of the mark relative to the reference position of the mark when the base point is set exceeds a preset threshold value, the estimation of the fastening location being operated is stopped.

9. The mixed reality device according to claim 8, wherein: After the estimation is stopped, when the base point is reset, the estimation of the fastening location being operated is restarted.

10. The mixed reality device according to claim 8 or 9, wherein: Repeatedly obtain the above movement amount, Whenever the movement amount is obtained, it is determined whether the movement amount exceeds the threshold value.

11. A processing method, wherein: Mixed reality devices are, Use the markers set in the real space to set the base point of the virtual space. In the three-dimensional coordinate system based on the base point, a virtual object is displayed at a preset position. obtaining the movement amount of the mark relative to the reference position of the mark when the reference point is set, When the movement amount exceeds a preset threshold value, the base point is moved according to the movement amount.

12. A program, wherein: Enable the mixed reality device to execute the processing method described in claim 11.

13. A storage medium, wherein: The program according to claim 12 is stored.

Images