Information display system

Abstract

Information display system (100) comprising: an imaging unit (110) configured to photograph a workpiece before and during processing and to create model data indicating the shape of the workpiece before and during processing, respectively; a complementation unit (120) configured to identify an invisible part of the workpiece based on a comparison of feature points extracted by the imaging unit (110) during processing with feature points of a previously held 3D model of the workpiece, and to generate a 3D model of the invisible part of the workpiece; and a display unit (140) configured to display the 3D model of the invisible part of the workpiece in a superimposed manner during machining.

Description

BACKGROUND OF THE INVENTIONFIELD OF THE INVENTIONThe present invention relates to an information display system, and more particularly, to an information display system configured to display additional information in a real space in a superimposed manner depending on the visibility of a workpiece.DESCRIPTION OF THE PRIOR ARTMixed reality (MR) technology and match move technology are known technologies. In MR technology, information on a virtual space is displayed in a real space in a superimposed manner. In the match move technology, a moving image is synthesized with other images. For example, JP 2002-269 546 A describes a system capable of automatically following the movement of a face of a person in a moving image and synthesizing an image obtained by reforming the face image into a desired shape. JP 2012-019 434 A describes a system capable of synthesizing an advertising video and a main video by a tracking technique.On the other hand, in the field of machine machining by machine tools, there arises a problem that it is difficult to visually recognize the state of a workpiece being machined (see FIG. 1 ) due to interference with a shielding object such as a cutting oil or a soiled window of a guard door. For example, there is a desire to visually recognize the state of machining progress of the workpiece, the occurrence of air cutting (unnecessary movement history of a tool), a size of machining tolerance (portion covered before the tool actually comes into contact with the workpiece after starting cutting feed), a ratio between the workpiece and chips, a tip position of the tool, and the like, if possible. Due to the above-described problem, it is currently difficult to recognize these peculiarities during machining. At present, therefore, the detection is made by temporarily interrupting the supply of the cutting oil or by slightly opening the guard door and looking through the gap formed. However, these methods are problematic in that safety is poor and time elapses until the field of view is better.Therefore, although it has been expected to solve these problems by using the above-described MR technology or match move technology, no specific solutions are disclosed in JP 2002-269 546 A and JP 2012-019 434 A.DE 10 2009 029 064 A1 discloses a device for monitoring the machining status with a multiplicity of CCD cameras for imaging the tool and the workpiece from different viewing angles. For each CCD camera, a virtual image is generated based on a 3D model of the workpiece and the tool, and a display unit allows a user to select either a virtual image in which a distal end of the tool is not hidden by the workpiece or the actual camera image.From "KÄCHOFEN, Denis; MENDEZ, Erick; ENGLZRNST, Diter: Interactive Focus and Context Visualization for Augmented Reality, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, 06.06.2008, pp. 191-201" a method for augmented reality display is known, in which invisible parts of a car are displayed superimposed on an image. Focus and context information is used to prevent the superimposed image from being over-sampled with unnecessary detail.Wikipedia, The Free Encyclopedia, May 26, 2016, describes the term "matching" as a film technique that enables computer graphics in live action images at the correct position, on the correct scale, in the correct orientation and movement with respect to the photographed objects in the image, and various methods can be used to extract camera movement information.SUMMARY OF THE INVENTIONThe present invention has been made in view of these problems, and its object is to provide an information display system configured to display additional information in a real space in a superimposed manner depending on the visibility of a workpiece. This object is achieved by an information display system according to claim 1.An information display system according to a specific embodiment of the present invention includes an imaging unit configured to photograph a workpiece and generate model data indicating the shape of the workpiece, a complementing unit configured to display a 3D model image of at least a part of the workpiece based on 3D model data of the workpiece and complement an invisible part of the workpiece by the 3D model image, and a display unit configured to display the 3D model image so as to be superimposed on the invisible part of the workpiece.An information display system according to another embodiment further includes a match move unit configured to perform match move processing for the model data and the 3D model image.In an information display system according to another embodiment, the complementing unit performs the complementing by comparing corresponding feature points of the model data and the 3D model.In an information display system according to another embodiment, the imaging unit creates model data of the workpiece and an environment of the workpiece, and the complementing unit performs complementation by using the model data of the environment.According to the present invention, there can be provided an information display system configured to display additional information in a real space in a superimposed manner depending on the visibility of a workpiece.BRIEF DESCRIPTION OF THE DRAWINGSThe above and other objects and features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings, in which: FIG. 1 is a diagram showing a problem of the related art; FIG. 2 is a schematic diagram showing a practical example of an information display system; FIG. 3 is a block diagram showing a configuration of the information display system; FIG. 4 is a flowchart showing an operation of the information display system; FIG. 5 is a schematic diagram showing an operation example of the information display system; and FIG. 6 is a schematic diagram showing an operation example of the information display system.DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTSAn embodiment of the present invention will now be described with reference to the accompanying drawings.FIG. 3 is a block diagram showing a configuration of an information display system 100 according to the embodiment of the present invention. The information display system 100 includes an imaging unit 110, complementing unit 120, match move unit 130, and display unit 140. The information display system 100 is an information processor typically including a central processing unit (CPU), storage device, and input and output devices such as an imaging device and a display device, and the various processing units thereof are logically implemented when the CPU executes a predetermined program. For example, the information display system 100 may be a smartphone, a tablet terminal, a head-mounted display (HMD), or the like, provided with a display configured to serve as the display unit 140 and in which the CPU and the storage device are incorporated. Alternatively, the information display system 100 may be constructed by connecting the display or the HMD serving as the display unit 140 to a computer provided with the CPU and the storage device.The imaging unit 110 acquires an image by photographing an object or the like existing in the real world, and executes processing for generating model data of the object included in the image, that is, data indicating the shape of the object. In the present embodiment, a moving image of a workpiece under work is captured and model data of the workpiece included in the moving image is created. The moving image is captured by a camera attached to the HMD, for example. Since the method of creating the model data from the image is well known, the detailed description thereof will be omitted. Typically, the model data to be generated is wire mesh data. Alternatively, it may be a simple set of feature points extracted from the image.The complementing unit 120 complements the model data generated by the imaging unit 110 by a 3D model of the workpiece and performs processing to cause the display unit 140 to display it. Specifically, the complementing unit 120 previously holds 3D model data (typically, CAD data or NC data) of the workpiece in a storage area (not illustrated) and performs reproduction so that the 3D model data is superimposed on the created model data based on an actual workpiece. In the present embodiment, if a part of the workpiece becomes invisible by a shielding object such as cutting oil or a guard door, 3D data of the invisible part is displayed.The match move unit 130 performs processing for associating the model data generated by the mapping unit 110 with the 3D model displayed by the complementing unit 120. Specifically, the match move unit 130 tracks the model data and sets the display position, direction, and the like of the 3D model. Thus, the 3D model to be complemented follows the workpiece even when the camera is moved to capture the workpiece, so that the state of the workpiece or the like can be continuously observed. Since the specific procedure of match move processing is well known, the detailed description thereof will be omitted.The display unit 140 is a display device that can visually recognize the actual workpiece and the 3D model generated by the complementing unit 120 in a superimposed manner. For example, the display unit 140 is an HMD equipped with a camera (see FIG. 2 ). In the case of a non-transmission type HMD, an image captured by the camera is displayed on a projection surface.In the case of a transmission type HMD, a field of view equivalent to the image captured by the camera can be visually recognized through a transparent projection surface. While the imaging unit 110 creates the model data based on the image captured by the camera, the model data does not need to be displayed on the display unit 140 and is held in a predetermined storage area and used for complementing processing and the match move processing. On the other hand, the display unit 140 displays the 3D model reproduced by the complementing unit 120 on the projection surface. In this way, a user can visually recognize the actual workpiece projected onto the projection surface or visually recognized by the projection surface and 3D data projected onto the projection surface in a superimposed manner.FIG. 4 is a flowchart showing an operation of the information display system 100.S1: Imaging processing (initial stage)First, the imaging unit 110 acquires an image of the workpiece before machining by using an image capturing means such as a camera. It is then preferable to acquire the image of the entire workpiece. The complementing processing and the match move processing in the following stage can be performed precisely without acquiring a large number of feature points on different parts of the workpiece beforehand.It is more preferable to acquire not only the image of the workpiece but those of its surroundings, for example, a table that supports the workpiece and the housing of a machine tool. Thus, even if the workpiece is not visually recognizable at all, the complementing processing and the match move processing can be performed in the following stage if the vicinity of the workpiece is visually recognizable. In general, the table, the housing of the machine tool, and the like are preferable surroundings to photograph because their positions relative to the workpiece under machining do not change. Otherwise, those surroundings to be photographed whose positions relative to the workpiece can be identified are preferable even if their relative positions are changeable. For example, although the position of a tool changes during machining, it may be identified from a machining program or the like at a certain time.The imaging unit 110 recognizes the workpiece in a photographed image and creates the model data of the workpiece. Typically, the imaging unit 110 extracts three-dimensional feature points individually from the previously held 3D model of the workpiece and the photographed image, and compares them. Then, it extracts and stores only those feature points in the image corresponding to the feature points of the 3D model. Preferably, the imaging unit 110 also extracts the feature points from the vicinity of the workpiece. Then, it stores a data set including the feature points of the workpiece and those of the environment.S 2: Imaging processing (during machining)Then, the imaging unit 110 acquires an image of the workpiece under machining using the image capturing means such as the camera, and extracts the feature points of the workpiece under machining in the same manner as in step S 1. Preferably, the imaging unit 110 also extracts the feature points from the vicinity of the workpiece.S3: Complementation ProcessingThe complementing unit 120 compares the feature points of the workpiece acquired in step S 2 and those of the previously held 3D model of the workpiece. If the comparison indicates that there is a part missing from the feature points of the workpiece acquired in step S 2, it can be assumed that this part is the part that becomes invisible by the shielding object.Then, the complementing unit 120 reproduces only the part of the 3D model corresponding to the invisible part of the workpiece. Specifically, it represents a 3D submodel including the feature points on the 3D model page corresponding to the feature points missing on the image page. The complementing unit 120 causes the display unit 140 to display the generated 3D submodel. Here, the complementing unit 120 identifies those feature points included in the feature points of the workpiece acquired in step S 1 but not included in the feature points of the workpiece acquired in step S 2. These are feature points that should be present in the invisible part. The complementing unit 120 compares the feature points that should be present in the invisible part with the feature points of the previously created 3D submodel and positions the 3D submodel so that their relative positions substantially coincide. Subsequently, the 3D submodel is displayed so that the invisible part of the actual workpiece is complemented (see FIG. 5 ).FIG. 6 is a diagram showing how a projection part of the 3D model formed by the complementing unit 120 changes depending on the conformity of the visibility of the workpiece, that is, the size of the invisible part of the workpiece. If the visibility of the workpiece is so high that there is no invisible part as illustrated on the left side of FIG. 6, the 3D model is not projected. If a part of the workpiece is missing as illustrated in the middle part of FIG. 6, the 3D submodel corresponding to the missing part is projected. If the workpiece is almost completely invisible, as shown on the right side of FIG. 6, the entire 3D model is projected.If the workpiece is almost completely invisible as shown on the right side of FIG. 6, or if the feature points that can be extracted from the image of the workpiece are so few that the position at which the 3D model is positioned cannot be identified, the complementing unit 120 may determine the position of the 3D model on the basis of the feature points of the vicinity of the workpiece. Specifically, the image of the workpiece including the environment is acquired in step S 1, so that the relative positions of the workpiece and the environment can be identified. Then, an image including the environment is also acquired in step S 2, and differences in feature points between this image and the image acquired in step S 1 are obtained. In this way, the position at which the feature points of the workpiece should be present can be identified. The complementing unit 120 positions the 3D model so that these positions are covered.S4: Match Move ProcessingThe match move unit 130 performs processing that causes the 3D model reproduced in step S 3 to be tracked by a conventional method according to the change in the position of the workpiece in the image.S5: Repetition for every predetermined timePreferably, the information display system 100 repeats the processing related to steps S 2 to S 4 for each predetermined time point. Then, the change in visibility of the workpiece may be tracked in real time to enable an appropriate superimposed display of the 3D model.According to the present invention, if a part or the whole of the actual workpiece becomes invisible by the shielding object such as the cutting oil or the guard door, the information display system 100 generates the 3D partial model corresponding to the invisible part and displays the generated 3D model so as to superimpose on the invisible part of the actual workpiece. Consequently, the state of machining progress of the workpiece, the occurrence of an air cut (unnecessary movement history of a tool), a magnitude of machining tolerance (portion covered before the tool actually comes into contact with the workpiece after start of cutting feed), a relationship between the workpiece and chips, a tip position of the tool, and the like can be recognized by the 3D model even in a state unsuitable for visual recognition.The present invention is not limited to the above-described embodiment and can be changed as appropriate without departing from the spirit of the invention. Each of constituent elements of the embodiment may be modified or omitted without departing from the scope of the present invention.For example, although in the above-described embodiment, the invisible part of the workpiece is predominantly replaced with the 3D model for display, the application intention of the present invention is not limited to the workpiece. For example, 3D models may also be created and displayed in a superimposed manner for the tool (tip portion thereof, in particular), a part of the machine tool, and the like.Further, when displaying the 3D sub-model on the actual workpiece in a superimposed manner, processing may be performed to increase the transparency of a part defining the boundary to the actual workpiece as shown in the lower middle portion of FIG. 6.Consequently, the actual workpiece and the 3D sub-model can be displayed more smoothly in a superimposed manner.Further, in the above-described embodiment, the complementing unit 120 identifies the invisible part of the workpiece in step S 3 by comparing the feature points of the workpiece acquired in step S 2 with those of the previously held 3D model of the workpiece. Alternatively, however, the invisible part of the workpiece may also be identified by comparing the feature points of the workpiece acquired in step S 2 and those of the workpiece acquired in step S 1. Specifically, it is only necessary that the feature points of the entire workpiece and those of the partially invisible workpiece can be compared. Further, although the example in which the HMD is used as the display unit 140 is mainly described in the above-described embodiment, the present invention is not limited thereto. For example, a display capable of displaying the moving image captured by the camera, a transmission type protective cover window capable of directly visually recognizing the workpiece and also projecting the 3D model, or the like may alternatively be used as the display unit 140.

Claims

An information display system (100) comprising: an imaging unit (110) configured to photograph a workpiece before and during machining and generate model data indicating the shape of the workpiece before and during machining, respectively; a complementing unit (120) configured to identify an invisible part of the workpiece based on a comparison of feature points extracted by the imaging unit (110) during machining with feature points of a previously held 3D model of the workpiece and to generate a 3D model of the invisible part of the workpiece; and a display unit (140) configured to display the 3D model of the invisible part of the workpiece during machining in a superimposed manner.The information display system (100) according to claim 1, further comprising a match move unit (130) configured to perform match move processing for the model data and the 3D model.The information display system (100) according to claim 1, wherein the imaging unit (110) creates model data of the workpiece and the vicinity of the workpiece before machining and model data of the vicinity during machining, and the complementing unit (120) identifies positions at which the feature points of the workpiece should be present using the model data of the vicinity during machining and positions the 3D model to be displayed in a superimposed manner so as to cover these positions.

Images