Refinement of three-dimensional models

Abstract

A method for refining a three-dimensional model comprising the steps of: providing a three-dimensional model of an object; obtaining a first two-dimensional image of the object from a first camera pose; obtaining a second two-dimensional image of the object from a second camera pose, wherein the second two-dimensional image shares a common section of a surface of the object with the first two-dimensional image; deforming the first two-dimensional image based on a spatial relationship between the first camera pose, the second camera pose, and the three-dimensional model to obtain an expected image from the second camera pose based on the first camera pose; comparing the second two-dimensional image with the expected image to identify one or more discrepancies.and correcting the three-dimensional model based on one or more discrepancies: wherein the first camera pose and the second camera pose have a position and orientation of a single camera in two independent positions, and wherein a relationship between the first camera pose and the second camera pose is calculated based on a three-dimensional measurement of the surface of the object from the first camera pose and the second camera pose, respectively.

Description

[0001] The present application claims priority from provisional US patent application No. 61 / 019,159, filed on January 4. 2008, which is hereby incorporated in its entirety by reference. Field of invention

[0002] This invention generally relates to a three-dimensional imaging and, in particular, the refinement of three-dimensional models, which can be reconstructed from a sequence of three-dimensional measurements taken along a camera path. background

[0003] In a technique for three-dimensional image reconstruction, a number of images or image sets of an object are combined with a The camera captures images as it moves along a path across the surface of the object. Information from this image catalog can then be used. to create a three-dimensional model of the object based on each camera position and along the path captured three-dimensional To reconstruct the measurement. While single measurements from the camera can contain noise from a variety of sources, the The resulting three-dimensional model compensates for this noise to produce three-dimensional data points more accurately than the individual measurements. to recover.

[0004] There remains a need for post-processing techniques to perform individual three-dimensional measurements based on the full data set. refine, which is available for a complete three-dimensional recording. Summary

[0005] A three-dimensional measurement is refined by taking two-dimensional images of an object from offset camera positions. according to a three-dimensional model of the object, and any resulting discrepancies are applied to to refine the three-dimensional model, or to refine one of a number of three-dimensional measurements used to generate the three-dimensional models are used.

[0006] In one aspect, a method described herein for refining a three-dimensional model has the following features: providing a three-dimensional model of an object; obtaining a first two-dimensional image of the object from a first camera pose; Obtaining a second two-dimensional image of the object from a second camera pose, where the second two-dimensional image has a exhibits a common section of the object's surface with the first two-dimensional image; deformation of the first two-dimensional Images based on a spatial relationship between the first camera pose, the second camera pose, and the three-dimensional model, in order to create a To obtain the expected image from the second camera pose based on the first camera pose; to compare the second two-dimensional image with the expected image to identify one or more discrepancies; and correcting the three-dimensional model based on one or more the several discrepancies.

[0007] The first camera pose and the second camera pose can define a position and orientation of a single camera in two dependent These are positions. The first camera pose and the second camera pose can represent a position and orientation of two offset channels. a camera with multiple apertures. The first camera pose and the second camera pose can indicate a position and orientation of a The single camera can be in two independent positions. A relationship between the first camera pose and the second camera pose can be established. based on a three-dimensional measurement of the object's surface from the first camera pose and the second camera pose The method can be calculated by deriving the three-dimensional model from multiple three-dimensional measurements of the surface. The object can be captured in multiple camera poses, including the first and second camera poses. The method can... Applying one or more discrepancies to directly refine the three-dimensional model. The procedure can involve applying which exhibit one or more discrepancies in order to obtain a three-dimensional measurement from the first camera pose and / or the second camera pose to refine in order to provide a more refined measurement. The procedure can refine a camera path calculation for a camera path. exhibiting, which is used to generate the three-dimensional model, with the refined measurement being used to create a refined camera to provide a path. The procedure may involve the use of refined camera path and refined measurement to provide the to refine a three-dimensional model. The three-dimensional model can be a point cloud or a polygonal grid. The object It could be a human dentition. The second camera pose could correspond to a central channel of a multi-aperture camera system. the central channel provides a conventional two-dimensional image of the object. The method can obtain a third two-dimensional image of the object from a third camera pose, which is a second side channel of the system with multiple apertures corresponds to, and the deformation of the third two-dimensional image into an expected image for the central channel for use in further Refining the three-dimensional measurement of the camera system with multiple apertures.

[0008] In another aspect, a computer program product for refining a three-dimensional model of an object, described herein is a computer-executable code contained on a computer-readable medium which, when placed on a or is performed by several computer devices, which performs the following steps: providing a three-dimensional model of an object; Obtaining a first two-dimensional image of the object from a first camera pose; Obtaining a second two-dimensional image of the object from a second camera pose, with the second two-dimensional image showing a common section of a surface of the exhibits the object with the first two-dimensional image; deformation of the first two-dimensional image based on a spatial Relationship between the first camera pose, the second camera pose, and the three-dimensional model to produce an expected image from the second camera pose based on the first camera pose; comparing the second two-dimensional image with the expected image to determine one or more To identify discrepancies; and to correct the three-dimensional model based on one or more discrepancies.

[0009] The first camera pose and the second camera pose can define the position and orientation of a single camera in two dependent Positions. The first camera pose and the second camera pose can represent a position and orientation of two offset channels of a The camera should have multiple apertures. The first camera pose and the second camera pose can determine the position and orientation of a single aperture. The camera can be in two independent positions. A relationship between the first camera pose and the second camera pose can be based on... a three-dimensional measurement of the object's surface is calculated from the first camera pose and the second camera pose, respectively. The computer program product can include code to execute the step of deriving the three-dimensional model from several three-dimensional measurements of the object's surface from multiple camera poses, including the first camera pose and the second exhibit camera pose. The computer program product may contain code to execute the step of applying one or more exhibit discrepancies in order to directly refine the three-dimensional model. The computer program product can include code to execute the Step of applying one or more discrepancies to obtain a three-dimensional measurement from the first camera pose and / or to refine the second camera pose to provide a more refined measurement. The computer program product can execute code to do this. the step of refining a camera path calculation for a camera path that is used to generate the three-dimensional model is used, whereby the refined measurement is used to provide a refined camera path. The computer program product may contain code to execute the step of using the refined camera path and refined measurement to achieve the to refine a three-dimensional model. The three-dimensional model can be a point cloud or a polygonal grid. The object can a human dentition. The second camera pose can correspond to a central channel of a multi-aperture camera system, whereby The central channel provides a conventional two-dimensional image of the object. The computer program product can execute code. the steps of obtaining a third two-dimensional image of the object from a third camera position, which is a second side channel of the Systems with multiple apertures correspond to, and the deformation of the third two-dimensional image to an expected image for the central channel. Use in further refining the three-dimensional measurement of the camera system with multiple apertures. Brief description of the drawings

[0010] The invention and the following detailed description of certain embodiments thereof can be found in the following The characters are understood.

[0011] Fig. 1 shows a three-dimensional recording system.

[0012] Fig. 2 shows a schematic diagram of an optical system for a three-dimensional camera.

[0013] Fig. 3 shows a processing pipeline for obtaining three-dimensional data from a video camera.

[0014] Fig. 4 represents a coordinate system for three-dimensional measurements.

[0015] Fig. 5 shows a sequence of images captured by a moving camera.

[0016] Fig. 6 is a conceptual representation of a three-dimensional data acquisition process.

[0017] Fig. 7 is a flow chart of a process for refining three-dimensional data.

[0018] Fig. 8 is a flowchart of a global path optimization. Detailed description

[0019] In the following text, references to details in the singular should be understood as including details in the plural and Conversely, unless explicitly stated otherwise or it is clear from the text, grammatical conjunctions are intended to... to specify all conceivable disjunctive and conjunctive combinations of adjacent clauses, sentences, words and the like, unless explicitly stated otherwise or it is clear from the context.

[0020] The following description presents specific recording technologies in detail and focuses on dental applications of a three-dimensional imaging; however, it will be recognized that variations, adaptations, and combinations of the following methods and Systems will be clear to a typical expert. While, for example, a system based on images can be described, systems based on images cannot be described on images. Image-based recording techniques such as infrared flight time techniques or structured lighting techniques using pattern projections in a similar way Use reconstructions based on a camera path that can utilize the improvements described herein. Accordingly. It will be understood that while the following description describes a refinement using parallel images from two offset channels of a The camera with multiple apertures highlights techniques that can be applied accordingly to capture data frame by frame for a To refine the camera path of a camera with multiple apertures, or different frames of data for a conventional camera. While the Digital dentistry is a useful application of the improved accuracy resulting from the techniques described herein, and can be considered Another example is that the teachings of this revelation can also be used to create three-dimensional animation models, three-dimensional to refine images for machine vision applications and so on. It is intended that all such variations, adjustments, and Combinations fall within the scope of this revelation.

[0021] In the following description, the term “image” generally refers to a two-dimensional set of pixels which to form a two-dimensional view of an object within an image plane. The term "image set" generally refers to A set of related two-dimensional images that could be resolved into three-dimensional data. The term "point cloud" Generally refers to a three-dimensional set of points that form a three-dimensional view of a structure composed of a number of two-dimensional objects. Images of the reconstructed object are formed. In a three-dimensional image acquisition system, a number of such point clouds can also be generated. are recorded and combined into a total point cloud, which is made up of images captured by a moving camera. Consequently It will be understood that pixels generally denote two-dimensional data and points generally denote three-dimensional data. unless another meaning is specifically stated or becomes clear from the context.

[0022] The terms “three-dimensional model”, “three-dimensional surface representation”, “digital Surface representation,” “three-dimensional surface map” and the like, as used herein, are for this purpose. defined as any three-dimensional reconstruction of an object, such as a point cloud of surface data, a set of two-dimensional To denote polygons or any other data that represent all or part of the surface of an object, as defined by the recording and / or processing of three-dimensional recording data could be obtained, unless otherwise expressly stated. determined or otherwise evident from the context. A “three-dimensional representation” can be any of the above. described three-dimensional surface representations, as well as volumetric and other representations, unless otherwise specified. explicitly stated or otherwise made clear from the context.

[0023] In general, the terms “image processing” or “image preparation” refer to a two-dimensional The visualization of a three-dimensional object, such as for display on a monitor. However, it will be understood that a variety three-dimensional image processing technologies are available and can be usefully used with the systems and methods disclosed herein can. For example, the systems and methods described herein can be usefully used for a holographic display, an autostereoscopic display, or a similar system. Display, an anaglyph display, a head-mounted stereo display, or any other two-dimensional and / or three-dimensional display The image processing described herein should generally be interpreted as such unless a narrower meaning is explicitly specified. or becomes clear from the context in some other way.

[0024] The term “dental object”, as used herein, is intended to generally refer to an object to denotes something related to dentistry. This can refer to intraoral structures, such as dentition, and more typically, a human dentition. such as individual teeth, quadrants, full arches, pairs of arches (which may be separate or include different types), Soft tissues and the like, as well as bones and any other supporting or surrounding structures, are included. As he is described herein. When used, the term “intraoral structures” refers to both natural structures within a mouth, as above. described, as well as artificial structures, such as any of the dental objects described below that might be present in the mouth. Dental items can include “restorations”, which are generally understood to include components, that restore the structure or function of an existing dentition, such as crowns, bridges, veneers, inlays, onlays, amalgams, Composites and various substructures such as caps and the like, as well as temporary restorations for use during a A permanent restoration is produced. Dental items can also include a "prosthesis," which replaces a dentition through Removable or permanent structures are replaced, such as dentures, partial dentures, implants, retained dentures, and the like. Dental Items may also include “apparatuses” used to correct, align, or to adjust temporarily or permanently in other ways, such as removable orthodontic appliances, surgical stents, bruxism appliances, Snoring protectors, indirect bracket placement appliances, and the like. Dental items can also be "hardware." include those that are attached to a dentition for an extended period of time, such as implant attachments, implant abutments, orthodontic appliances Brackets and other orthodontic components. Dental items can also be "temporary components" of the dental procedure. Manufacturing includes (complete and / or partial) dental models, wax models, covering molds and the like, as well as shells, bases, Matrices and other components used in the manufacture of restorations, prostheses, and the like. Dental items They can also be natural dental objects, such as teeth, bones, and other intraoral structures described above, or They are categorized as artificial dental objects, such as restorations, prostheses, appliances, hardware and temporary components of the dental manufacturing, as described above.

[0025] Terms such as “digital dental model”, “digital dental impression” and the like are intended to three-dimensional representations of dental objects are used in various aspects of recording, analysis, prescription and The term "manufacturing" can be used unless another meaning is otherwise determined or becomes clear from the context. Expressions such as “Dental model” or “dental impression” are intended to denote a physical model, such as a cast, A printed or otherwise manufactured physical example of a dental item. Unless specifically stated otherwise, the The term “model”, when used on its own, refers to both a physical model and a digital model.

[0026] It will further be understood that terms such as “tool” or “control”, when used, To describe aspects of a user interface, these terms are generally intended to denote a variety of techniques that are used in a graphical user interface or other user interface to receive user input that initiates or controls a processing operation, including without limitation drop-down lists, radio buttons, pointer and / or mouse actions. (Point selection, area selection, drag-and-drop operations, etc.), selection boxes, command lines, text input fields, messages and alarms, progress bars, and so on. A tool or control can also include any physical hardware that User input includes devices such as a mouse, keyboard, display, keypad, trackball and / or any other device that provides a Receives physical input from a user and converts that physical input into an input for use in a computer system converts. Consequently, the following description should use the terms "tool", "control", and the like. It is to be understood generally unless a more specific meaning is determined in another way or becomes clear from the context.

[0027] Fig. 1 shows a three-dimensional recording system used with the systems and methods described herein. can be. In general, the system 100 can have a camera 102 that takes pictures of a surface 106 of an object. 104 records the images of a dental patient and forwards them to a computer 108, which displays a screen 110 and one or more User input devices 112, 114 such as a mouse 112 or a keyboard 114 may be included. The camera 102 may also have a integrated input or output device 116 such as a control input (e.g. a button, a touch panel, a thumbwheel, etc.) or a Have a display (e.g., LCD or LED display) to provide status information.

[0028] The camera 102 can comprise any camera or camera system suitable for capturing images from which a Three-dimensional point clouds or other three-dimensional data can be recovered. For example, camera 102 can be used to create a system comprising multiple apertures, as disclosed in U.S. Patent No. 7,372,642 by Rohály et al., the entire contents of which are incorporated herein by reference. While Rohály discloses a system with multiple apertures, it will be recognized that any system with multiple apertures that suitable for reconstructing a three-dimensional point cloud from a number of two-dimensional images, it can be used accordingly. can. In an embodiment with multiple apertures, the camera 102 can have multiple apertures, including a central aperture which is arranged along an optical central axis of a lens which provides a central channel for the camera 102 together with any associated provides imaging hardware. In such embodiments, the central channel can display a conventional video image of the recorded object. provide, while a number of axially offset channels yield image sets containing disparity information, which is used in three-dimensional Surface reconstruction can be used. In other embodiments, a separate video camera and / or a A channel will be provided to achieve the same result, i.e., a video of an object temporarily resembling a three-dimensional recording. of the object, preferably from the same perspective, or from a perspective with a fixed known relationship to the Camera 102 perspective. Camera 102 can also, or instead, be a stereoscopic, triple, or other multiple camera. or other configurations in which a number of cameras or optical paths are held in a fixed relationship to each other to obtain two-dimensional images of an object from a number of different perspectives. The Camera 102 can be have suitable processing capabilities for deriving a three-dimensional point cloud from a set or number of image sets, or Each two-dimensional image set can be transferred to an external processor, such as the one described in Computer 108 below. is included. In other embodiments, the camera 102 can capture structured light, a laser image, a direct distance measurement, or use any other technology suitable for capturing three-dimensional data or two-dimensional data that can be used to create three-dimensional models Data can be resolved. While the techniques described below make sense in using video data, which is captured by a video-based system. Since the three-dimensional recording system is based on this, it will be understood that any other three-dimensional recording system can be supplemented with a video recording system that simultaneously or otherwise captures suitable video data along with the recording of records three-dimensional data synchronously.

[0029] In one embodiment, the camera 102 is a freely positionable handheld probe with at least one user input device. 116 , such as a button, a lever, a dial, a knurled wheel, a switch or the like for user control of the image acquisition system 100, such as starting and stopping recordings. In one embodiment, the camera 102 can be used for Dental imaging must be shaped and dimensioned. In particular, camera 102 can be shaped and dimensioned for intraoral imaging and data acquisition. be measured as by inserting into the mouth of an imaging object and guiding over an intraoral surface 106 in a suitable distance to capture surface data of teeth, gums, and so on. The camera 102 can be used by such a continuous data acquisition process, a point cloud of surface data with sufficient spatial resolution and accuracy to capture dental items such as prosthetics, hardware, appliances and the like, either directly or through a variety of methods. to produce intermediate processing steps. In other embodiments, surface data from a dental model such as a dental prosthesis can be obtained. to be recorded in order to ensure proper adaptation by means of a prior recording of the corresponding dentition, such as a for the Prosthesis prepared tooth surface.

[0030] Although not shown in Fig. 1, it will be recognized that a number of supplementary lighting systems are usefully used during the Image capture can be used. For example, the ambient lighting can be enhanced with one or more spotlights, which Illuminate object 104 to accelerate image acquisition and improve depth of field (or spatial resolution). Camera 102 can also, or instead, use a strobe light, a flash, or another function. Provide a light source to supplement the illumination of object 104 during image capture.

[0031] The object 104 can be any object, an assembly of objects, a section of an object or other content of an object. In particular, object 104, with regard to the dental techniques discussed herein, can be a human dentition. encompasses images captured intraorally from within the mouth of a dental patient. An image can be a three-dimensional representation of a part or to capture the entirety of the dentition according to a specific purpose of the recording. Consequently, the recording can be a digital model of a tooth, a quadrant of teeth or a full set of teeth including two opposing arches, as well as soft tissue or any other relevant intraoral structures. The image can capture multiple views, such as a tooth surface. Before and after preparation for a restoration. As shown below, this data can be used for subsequent modeling. This will be done, for example, to design a restoration or to determine a boundary line for it. During the recording, a central canal of the Camera 102 or a separate video system captures a video of the dentition from the viewpoint of camera 102. In other embodiments, where For example, if a finished product is virtually adapted to a surface preparation on a trial basis, the recording can be a dental prosthesis. such as an inlay, a crown or any other dental prosthesis, dental hardware, dental appliances or the like. Item 104 Alternatively, or in addition, a dental model such as a plaster cast, a wax cast, an impression, or a negative impression of a tooth, teeth, soft tissue or a combination thereof.

[0032] The computer 108 can, for example, comprise a personal computer or other processing equipment. In one embodiment The Computer 108 comprises a personal computer with a dual 2.8 GHz Opteron central processing unit, 2 gigabytes of random access memory, a TYAN Thunder K8WE motherboard and a 250 GB, 10000 RPM hard drive. In a current embodiment, the The system is operated to capture more than five thousand points per image set in real time using the techniques described herein and to store a total point cloud of several million points. Of course, this point cloud can be further processed to... to adapt the subsequent data processing, such as by decimating the point cloud data or generating a suitable grid. of surface data. As used herein, the term "real-time" generally means without a perceptible Delay between processing and display. In a video-based recording system, real-time specifically refers to the Processing within the time between frames of video data, which according to the specific video technologies is between approximately fifteen Frames per second can vary and can range from about thirty frames per second. More generally, the processing capabilities of the computer can vary. 108 according to the size of the object 104 , the speed of image acquisition and the desired spatial resolution of the three-dimensional points vary. The Computer 108 can also use peripherals such as a keyboard 114, a display 110, and a Mouse 112 enables user interaction with camera system 100. Display 110 can be a touchscreen display. be capable of accepting user input through direct, physical interaction with display 110. In another aspect The display may include an autostereoscopic display or the like, capable of displaying stereo images.

[0033] The communication between the computer 108 and the camera 102 can use any suitable communication link, including, for example, a wired connection or a wireless connection, such as one based on IEEE 802.11 (also known as wireless). Ethernet (known as Ethernet), Bluetooth or any other suitable wireless standard that uses, for example, radio frequency, infrared or other Wireless communication medium used. In medical imaging or other sensitive applications, a wireless communication medium may be used. Image transmission from camera 102 to computer 108 is ensured. Computer 108 can generate control signals for camera 102. which, in addition to image capture commands, may include conventional camera controls such as focus or zoom.

[0034] In an example of the usual operation of a three-dimensional image acquisition system 100, the camera 102 can capture two-dimensional images. Capture image sets at a video rate while the camera is moved across the surface of the object. The two-dimensional image sets can be forwarded to computer 108 for the derivation of the three-dimensional point clouds. The three-dimensional data for each new The captured two-dimensional image set can be derived using a number of different techniques and linked to existing three-dimensional data. adapted or "added on". Such a system can use camera motion estimation to determine the need for a to avoid independent tracking of the position of camera 102. A useful example of such a technique is given in the jointly held US application no. 11 / 270,135, filed on November 9, 2005, the entire contents of which is included herein by reference. However, it will be recognized that this example is not limiting, and that the herein The described principles can be applied to a wide range of three-dimensional image acquisition systems.

[0035] The display 110 can include any display suitable for video or image processing at a different rate on a resolution level is suitable, one that corresponds to the acquired data. Suitable displays include cathode ray tube displays, liquid crystal displays, Light-emitting diode displays and the like. In general, the display 110 can be operationally coupled with the computer 108 and It must be capable of receiving display signals from it. This display can be a CRT or flat-screen monitor, a three-dimensional display. (such as an anaglyph display), an autostereoscopic three-dimensional display, or any other suitable two-dimensional or Three-dimensional image processing hardware is included. In some embodiments, the display may have a touchscreen interface. exhibiting, for example, capacitance, resistance, or acoustic surface wave signals (also known as dispersion signals) Touchscreen technologies or any other suitable technology for sensing the physical interaction with the display 110 used.

[0036] The system 100 can comprise a computer-usable or computer-readable medium. The computer-usable medium 118 can one or more memory chips (or other chips such as a processor that includes memory), optical disks, magnetic disks or other magnetic media, and so on. The computer-usable medium 118 can, in various embodiments, have a removable storage (such as a USB device, a tape drive, an external hard drive, and so on), remote storage (such as via a network) connected storage), volatile or non-volatile computer memory, and so on. The computer-usable medium 118 can The computer-readable instructions for execution by Computer 108 are included to carry out the various processes described herein. The computer-usable medium 118 can also, or instead, store data received from the camera 102, a three-dimensional Store the model of item 104, store the computer code for image processing and display, and so on.

[0037] Fig. 2 shows an optical system 200 for a three-dimensional camera, which is equipped with the systems and methods described herein. can be used as for the camera 102 described above with reference to Fig. 1 .

[0038] The optical system 200 can include a primary optical device 202 which can be used in any type of image processing system can. In general, a primary optical device here refers to an optical system with one optical channel. Typically, it uses This optical channel shares at least one lens and has a shared image plane within the optical system, although Variations of this are explicitly described in the following description or become clear from the context in other ways. The visual System 200 can be a single primary lens, a group of lenses, an objective, mirror systems (including conventional mirrors, digital mirrors) mirror systems, digital light processors or the like), confocal mirrors and any other optical devices that are suitable for use with the systems described herein. The optical system 200 can, for example, be used in a stereoscopic or Other multi-image camera systems can be used. Other optical devices may include holographic optical elements or the like. exhibit. In various configurations, the primary optical device 202 can have one or more lenses such as an objective (or a group). of lenses) 202b, a supplementary lens 202d, a relay lens 202f, and so on. The objective 202b can be located at or near an entrance pupil. The auxiliary lens 202d may be arranged on or near a first image plane 202c of the optical system 200. The relay lens 202f can direct bundles of light rays within the optical system 200. The optical system 200 can also include components such as aperture elements 208 with one or more apertures 212, a refocusing device 210 with a or several refocusing elements 204, one or more scanning devices 218 and / or a number of sensors 214a, 214b, 214c exhibit.

[0039] The optical system 200 can be designed for active wavefront scanning, which should be understood as encompassing any technique includes, which is used to scan a series or collection of optical data from an object 220 or objects, including optical data used to detect two-dimensional or three-dimensional properties of the object 220 using optical data to detect movement, using optical data for speed measurement or object tracking, or to help with such matters. Further details of an optical system that can be used as the optical system 200 of Fig. 2 are given in the US Patent No. 7,372,642 provided represents, the entire content of which is incorporated herein by reference. More generally, it will be understood that, while Fig. 2 a The embodiment of an optical system 200 represents numerous variations. A prominent feature of the optical system is The explanation below concerns the use of an optical center channel that transmits conventional video or still images using one of the sensors. 214b records data simultaneously with different offset data (e.g., 214a and 214c), which are used to perform three-dimensional measurements. Capture. This central channel image can be displayed on a user interface for inspection, marking, and other manipulation. to enable a user during a user session, as described below.

[0040] Fig. 3 shows a three-dimensional reconstruction system 300 which includes a high-speed pipeline and a high-precision pipeline The high-speed processing pipeline 330 is generally intended to provide three-dimensional data in real time, such as... a video frame rate used by an associated display, while the high-precision processing pipeline 350 aims to achieve the highest To provide the accuracy possible from camera measurements, dependent on any external calculation or time constraints that imposed by the system hardware or an intended use of the results. A data source 310 such as the one described above Camera 102 delivers image data or the like to system 300. Data source 310 can, for example, be hardware such as LED ring lights, Pen sensors, an image digitizer, a computer, an operating system and any other suitable hardware and / or software to obtain of data used in a three-dimensional reconstruction. Images from data source 310, such as mid-canal images, which contain conventional video images, and side channels containing disparity data used to provide depth information To recover data, it can be sent to the Real-Time Processing Controller 316. The Real-Time Processing Controller 316 can also Provide camera control information or other feedback to data source 310, which is used in subsequent data acquisition or for Specifying data that is already obtained in data source 310, which is processed by the real-time processing controller 316 are required. Full-resolution images and related image data can be stored in a full-resolution image storage 322. Stored images can, for example, be delivered to the high-precision processing controller 324 during processing, or kept for image review by a human operator during subsequent processing steps.

[0041] The real-time processing controller 316 can send images or frames to the high-speed (video rate) processing pipeline 330 for Reconstruct three-dimensional surfaces from two-dimensional source data in real time. In an exemplary embodiment. Two-dimensional images from an image set, such as side-channel images, can be registered by a two-dimensional image registration module 332. Based on the results of the two-dimensional image registration, a three-dimensional point cloud generation module can be created. 334. Create a three-dimensional point cloud or other three-dimensional representation. The three-dimensional point clouds are made up of individual Image sets can be combined using a three-dimensional attachment module 336. Finally, the attached measurements can be processed by a The three-dimensional model generation module 338 can be combined to form an integrated three-dimensional model. The resulting model can be used as A three-dimensional high-speed model 340 can be stored.

[0042] The high-precision processing controller 324 can supply images or frames to the high-precision processing pipeline 350. Two-dimensional image registration can be performed on separate image sets using a two-dimensional image registration module 352. Based on the results of the two-dimensional image registration, a three-dimensional point cloud or other three-dimensional data can be generated. The representation is generated by a three-dimensional point cloud generation module 354. The three-dimensional point clouds are composed of individual Image sets can be linked using a three-dimensional attachment module 356. A global motion optimization, which also includes this, referred to as global path optimization or global camera path optimization, can be achieved through a global motion optimization module 357. to reduce errors in the resulting three-dimensional model 358. In general, the path of the camera, since it the The image frame is calculated as part of the three-dimensional reconstruction process. In a post-processing refinement procedure, The calculation of the camera path can be optimized – that is, the error accumulation along the length of the camera path can be reduced by a Supplementary frame-to-frame motion estimation can be minimized with some or all global path information. Based on global Information such as individual data frames in the image memory 322, the three-dimensional high-speed model 340 and intermediate results The high-precision model 370 can be processed in the high-precision processing pipeline 350 to correct errors in the camera path and to reduce resulting artifacts in the reconstructed model. As a further refinement, a grid can be applied to the high-speed model. projected through a 360° grid projection module. The re The resulting images can be warped or deformed using a warping module 362. Warped images can be used to to facilitate alignment and joining between images, such as by reducing the initial error in motion estimation. Warped images can be supplied to the two-dimensional image registration module 352. The feedback from the three-dimensional The high-precision model 370 can be repeatedly inserted into the pipeline until a certain metric is obtained, such as an attachment accuracy or a minimal error threshold.

[0043] Various aspects of the system 300 of Fig. 3 are described in more detail below. In particular, a A model refinement process is described that can be used by the 324 high-precision processing controller to... to refine the three-dimensional high-precision model 370 using measured data in the image memory 322. It should be understood that The various processing modules or the steps involved in the modules, shown in this figure, are exemplary in their essence. are and that the processing sequence or the steps of the processing sequence are modified, omitted, repeated, rearranged or can be supplemented without leaving the scope of this revelation.

[0044] Fig. 4 shows a coordinate system for three-dimensional measurements, wherein a system such as the optical system described above 200 is used. The following description is intended to provide useful context and should in no way be considered restrictive. can be interpreted. In general, an object 408 within an image plane 402 of a camera has world coordinates {Xw, Yw, Zw} in a World coordinate system 410 , camera coordinates {Xc, Yc, Zc} in a camera coordinate system 406 , and image set coordinates {xi, yi, di(xi, yi)} for i = 1 to N points or pixels within a processing grid of the field of view 402, where is a disparity vector 412 which represents one or contains several disparity values ​​that represent the z-axis shift (Zc) or depth 404 of a point in the image plane 402 based on the x-axis and / or y-axis shift in the image plane 402 between a number of physically offset apertures or other imaging channels characterizes. The processing grid can be understood as any overlay or grid on an image or other two-dimensional data. These are the locations where processing will take place. While a processing grid is a regular grid of locations in a The processing grid can be square, rectangular, triangular or other patterns, or it can be irregular instead. These patterns are either randomly selected or chosen according to the specific item being processed. The disparity vector 412 This can be expressed, for example, as a shift relative to a center channel, if present, for the camera. Generally, it is encoded The disparity vector 412 represents the depth, and in various other three-dimensional imaging systems, this disparity vector 412 can be represented by One or more other measured quantities that encode the depth are replaced. Consequently, expressions such as disparity vector, disparity value, and other factors should be used. and disparity data and the like are generally understood to include one or more scalar and / or vector quantities, which are measured by a system to capture depth information. More generally, a three-dimensional measurement can also be used. as used herein, denotes any form of data that encodes three-dimensional data, including without limitation groups of two-dimensional images from which disparity vectors could be obtained, the disparity field (of disparity vectors) itself, or A three-dimensional surface reconstruction derived from the disparity field. In the case of an image-based three-dimensional For reconstruction, a camera model can be used to relate disparity vectors to the depth within a camera's field of view. to establish. The camera model can be theoretically based on optical modeling or other physics, or empirically through observation. or a combination thereof, and can be calibrated to correct optical aberrations, lens defects, and all other physical aberrations. To compensate for variations or characteristics of a particular physical system.

[0045] While a single image plane 402 is shown for illustrative purposes, it will be recognized that a camera with multiple Apertures (or other multi-channel systems) can have a number of physically offset optical channels, each with a provide a different image plane, and the differences in feature locations (the xy-shift) between the images for each optical channel can be described as the The disparity field can be represented. In various specific processing steps, the disparity data can be reduced to a single image layer, such as... a central channel image plane of the camera.

[0046] Fig. 5 shows a sequence of images captured by a movable camera. In the sequence 500, a camera 502, which for example, any of the cameras 102 described above may include an image of an object 504 from a number of different Positions 506a–506e are captured along a camera path 507. The camera path 507 is a continuous curved path. As depicted, representing the physical path of a camera, it will be understood that the camera Path 507 can be analytically represented by discrete, straight line transformations together with associated rotations in three-dimensional space. While five camera positions are shown in camera path 507 of Fig. 5, it will be recognized that in accordance with The principles described herein can be used for more or fewer camera positions. In one embodiment, the camera 502 in each position 506 a set of images: capture two-dimensional images from which a point cloud can be reconstructed (or any other suitable three-dimensional measurement for the camera position). In general, these three-dimensional point clouds (or other three-dimensional data), which are captured from the sequence 500, to a three-dimensional model such as a composite point cloud or another three-dimensional model of the object, such as by minimizing errors in a three-dimensional registration of individual three-dimensional measurements, or any combination of various other techniques. It should also be understood that In certain embodiments, the camera can remain stationary while the object moves. In such cases, rather the movement of object 504 is determined as the movement of camera 502, although the use of camera movement is compared to the Object movement is a relatively arbitrary matter of expediency or computational efficiency of a camera coordinate system. compared to an object coordinate system.

[0047] Fig. 6 is a conceptual representation of a three-dimensional data acquisition process 600, which is used in the systems described above. can be used. In general, the camera (which can be any of the cameras described above) receives two-dimensional measurements. a surface of an object 601 such as a first measurement 602 from a side channel (e.g. a left channel image), a second measurement from another side channel 604 (e.g. a right channel image), and a third measurement from a center channel 603 (e.g. a center channel image). It will be understood that while three channels are represented, a system can process three-dimensional data from more or fewer channels using various techniques can be recovered that will be clear to a typical professional, and it is intended that all such techniques will be able to be recovered. Techniques that can be improved using the refinement techniques described herein fall within the scope of this disclosure. Measurements 602, 603, 604 can, for example, be processed to obtain a disparity field 606 that represents the relative motion of Features within the images of each measurement were identified. A camera model 610 can be used to measure the disparity field 606. to relate to the three-dimensional reconstruction 612 of the surface of the object 610, which was measured from a camera pose will be. While a central canal image is expediently used as the reference for the camera pose of the resulting three-dimensional reconstruction 612 While it can be used, this is not necessary and may not always be available as a reference in certain systems. Three-dimensional reconstruction 612 can be linked to other such three-dimensional measurements using camera path information or the like. to be attached in order to obtain a three-dimensional model 620 of the item 601.

[0048] In a model refinement process described below, one of the two-dimensional measurements, such as the first measurement, can be 602 , onto the three-dimensional model using available spatial information (e.g. camera position and orientation). The resulting projection can then be back-projected onto the second camera pose using warping or other deformation techniques. This is necessary to obtain an expected measurement at the second camera position. In the case of a two-dimensional side-channel image or similar, The expected measurement can be a corresponding image expected in the central channel or another side channel. By adjusting the three-dimensional measurement from this image pair to eliminate an error between the actual and expected measurements in an overlapping To reduce or minimize the area of ​​the object, the three-dimensional measurement can be refined for that camera position. to represent a surface of object 601 more accurately. In one aspect, the three-dimensional model can be enhanced with the new spatial Information can be refined directly. In another aspect, the improved three-dimensional measurement for the camera can be used in a new way. Motion estimation is used to determine the camera path and three-dimensional model data for an entire shot or a section. to recover from it. By refining the individual three-dimensional measurements and the camera path in this way, a more accurate result can be achieved. three-dimensional model for The object will be received. It will be recognized that, in general, error minimization is achieved through a number of different processes. Data sets that encode three-dimensional information, such as two-dimensional image sets, or processed data sets can be used. Representations of this measurement, such as the disparity field.

[0049] Fig. 7 is a flowchart of a process for refining three-dimensional data. In general, the process 700 refines a three-dimensional model by refining individual three-dimensional measurements taken at different camera positions, This information can be used to refine the resulting model. This process 700 can be usefully used for... For example, it can be used to warp two-dimensional images in a module 362 of a high-precision processing pipeline 350, e.g. to improve the two-dimensional registration and / or the three-dimensional attachment results. While a particular embodiment As described in detail below, it will be recognized that some similar technique for generating an expected two-dimensional Measurement at one camera position using an actual two-dimensional measurement from another camera position together with a three-dimensional model of the depicted object (and a camera model if suitable) can be used similarly. Consequently The techniques described herein can easily be adapted to other systems that create a three-dimensional model from a series of individual three-dimensional measurements are obtained, such as systems that use structured light, or systems that generate a series of two-dimensional images. use.

[0050] As shown in step 710, process 700 can be used to acquire frames of image data along a camera path. This image data image pairs can include an image from two or more offset optical channels of a camera with multiple apertures or a other multi-channel imaging device. In one embodiment, each image in an image pair contains a two-dimensional image. consisting of two coupled poses with a known, fixed relationship to each other. In such an embodiment, a central channel can also be included. It is intended that a third image will be included as part of the image data frame to provide a conventional, undistorted two-dimensional view of the to provide information about the recorded object (where distortions in the side channels encode the distance to a surface). The center channel It can also serve as a reference pose for a three-dimensional measurement derived from the pair of two-dimensional images. However, it should be understood that this arrangement is somewhat arbitrary, and other cameras can be used, such as a camera with a central channel. and a single side channel, or just two side channels, or any number of other arrangements. More generally, any Camera that captures two-dimensional images for use in a three-dimensional reconstruction, using the techniques described herein be used.

[0051] As shown in step 712, three-dimensional measurements can be obtained from the image data. In general, this can the processing of image sets or the like to obtain disparity data via a camera processing grid, and furthermore a Processing the disparity data includes obtaining a three-dimensional surface reconstruction. In one embodiment, the Disparity data provides depth information and can be used to perform three-dimensional measurements using a camera model or similar device. to recover the data in order to relate disparity data with depth information for each pixel of the processing grid. This step The 712 process can be repeated for each individual measurement (e.g., image set) obtained by the camera. The result can be a three-dimensional image. Measurement or reconstruction can be obtained for each camera pose along a camera path. It will be understood that the disparity data itself three-dimensional measurement, and for many of the processing steps described herein instead of three-dimensional reconstruction can be used, with suitable adjustments easily understood by a typical professional. It will further be understood that that other three-dimensional imaging techniques are known and can be adapted to obtain three-dimensional measurements of an object surface.

[0052] As shown in step 714, a three-dimensional model can be created from the individual three-dimensional measurements obtained in step 712. can be built up. Where the three-dimensional measurements of the object's surface overlap, these three-dimensional Measurements can be recorded using any of a variety of known techniques. As a result, the camera path of Pose can be determined. can be recovered to pose, and the three-dimensional measurements from each pose can be used to create a full three-dimensional model of the The recorded areas of the object's surface are combined.

[0053] As shown in step 716, a two-dimensional image or other measurement from a channel of a camera can be spatially projected onto the Step 714: The fully obtained three-dimensional model is projected. In general, the unprocessed camera measurement comprises a two-dimensional image of pixel values, which are then applied to the three-dimensional model using a Texture mapping or any other suitable techniques can be projected to convert the two-dimensional data from the image sets into to use the coordinate system of the three-dimensional model. A significant advantage of this approach is the use of a three-dimensional model of the The subject matter may include, for example, global information that was not available when the data was initially collected. The model can, for example, average errors and / or reduce noise in individual camera measurements, as well as errors in a global measurement. Minimize camera path where possible. Using this initial model as a spatial reference point, process 700 can be applied to the individual three-dimensional measurements will be returned, as described further below.

[0054] As shown in step 718, the projected measurement from the three-dimensional model can be back-projected onto another channel of the camera. This can be the center channel or another side channel of the camera described above. The projected result from step 716 can be back-projected using any suitable techniques to create a synthetic view of the measurement from one camera channel. obtained as it should appear from the other camera channel, based on the spatial relationship between the projected result, the three-dimensional model and the position and rotation of the other channel. It will be recognized that if there are no errors p in the If the initial measurement were given, this synthetic view would correspond exactly to the actual two-dimensional image that would be derived from the another channel is received. However, in a high-speed processing pipeline like the one described above, it can happen that a The initial three-dimensional model cannot accurately capture surface details for several reasons (low-resolution processing, lack of global surface data, such as the complete three-dimensional model, etc.). Consequently, it is expected that in a practical system it will be Variations between a synthesized view (based on observations from another position) and an actual view can. For example, backprojection can be achieved by warping or otherwise deforming the projected result based on the Three-dimensional models and camera pose information are used for the respective measurements. This is achieved by processing these synthesized data. Image sets to obtain disparity data, and further back-projection of the synthesized disparity data by the camera model can be a The result is obtained by backprojection, which is a synthesized or expected version of the three-dimensional measurement from the second camera position. represented.

[0055] As shown in step 720, a three-dimensional measurement can be performed by a camera (e.g., the measurement taken from a set of images in (derived from a data frame) can be refined by adjusting the three-dimensional reconstruction to eliminate an error between the in the The back-projected result obtained in step 718 and a corresponding actual two-dimensional measurement recorded in step 710 are compared. minimize. More generally, where two images are taken from an overlapping section of the object's surface, measurements can be minimized. This includes a feature that allows for a direct comparison of the back-projected (e.g., synthesized) measurement and the actual measurement. In one embodiment, this can be achieved by... Camera calibration data and other information descriptive of the camera or the camera's channels are projected into the projection and / or Back projection can be incorporated to improve the three-dimensional accuracy of the resulting three-dimensional measurement.

[0056] As shown in step 722, the three-dimensional model can be based on the refined three-dimensional measurements for each Image data frames can be refined. A number of techniques can be used to refine the model. In one aspect The three-dimensional data can be used for a refined three-dimensional measurement to directly create the three-dimensional model. modify, e.g. by estimating the distribution of changes in the refined three-dimensional measurement during the reconstruction process for the three-dimensional model. In another aspect, a new motion-based reconstruction can be used for some or all of the Recording data is carried out using refined three-dimensional measurements instead of the initial three-dimensional measurements. to recover a camera path that is used to relate the individual measurements to a global coordinate system In another aspect, this process can be repeated to obtain an iterative refinement of the three-dimensional model, e.g. For example, for a predetermined number of iterations, or until a predetermined error threshold is reached, or until no further refinement is possible from a The previous iteration is obtained, and so on, as well as various combinations thereof. Iterations can be local (e.g., at specific points). areas where errors are large) or globally (e.g. for any overlapping area between camera positions) or in a combination thereof be performed.

[0057] It will also be recognized that this approach can be usefully combined with other three-dimensional reconstruction techniques, as well as on Other methods can be used within the image-pair-based processing described above. For example, while the one based on a Model-based refinement Since the accuracy of a specific three-dimensional measurement can be improved by using the same approach to... to project a two-dimensional image from one image set back onto a two-dimensional image from another image set, in order to create a frame-to-frame to achieve improvements in accuracy. Furthermore, these image sets can be offset by any number of intervening image sets, and complementarily, a bidirectional refinement can be performed on any and all of the preceding ones, wherever the two Measurements involve a certain degree of overlap on the object's surface. While the above technique is for testing a specific... As described in the set of overlapping measurements, more generally speaking, this technique can be used with any frequency, in any order, for some or All of the overlapping areas are repeated in measurements that are used to obtain a three-dimensional model, and it will intended that all such variations fall within the scope of this revelation.

[0058] Fig. 8 is a flowchart of a global path optimization. In one aspect, the refinement of individual three-dimensional measurements in This can be combined with numerical techniques for global path optimization for an entire camera path to perform a further iterative process. to achieve an improvement in the resulting three-dimensional model. A suitable global path optimization technique will now be discussed in more detail. Details described.

[0059] Process 800 can begin with preprocessing, as shown in step 810. It will be understood that the preprocessing, as As described herein, it presupposes the availability of a number of image data frames from which a camera path and a three-dimensional The model can be reconstructed. The information for the three-dimensional reconstruction can be generated in numerous ways, including from a structured light projection, a shading-based three-dimensional reconstruction, or Disparity data originates from a single source. Disparity data can be generated by a conventional image plus one or more additional channels or side channels. Preprocessing can determine the number of available frames, the amount of overlap between adjacent frames, the detection and removal of frames with blurry or highly distorted images, and all other suitable preprocessing steps This includes an initial estimate of the number of desired keyframes during the preprocessing step.

[0060] As shown in step 812, key frames can be selected from among all frames of data that are scanned by an image scanner. The data is captured along a camera path. Generally, computation costs can be reduced by storing certain data. certain calculations and processing steps will be performed exclusively in relation to keyframes. Key frames can be related to each other in a way that characterizes a complete camera path. This is made possible, typically, by registering overlapping three-dimensional data in respective keyframes. There are various... Techniques for selecting a subset of data frames known as key frames include techniques derived from a Image overlap, camera path, the number of intervening non-keyframes, and so on are all factors. Keyframes can in addition or instead based on an amount of image overlap from the previous keyframe and / or a candidate for A subsequent keyframe (if available) can be selected. Even a small overlap can affect frame-to-frame registration. Impairment. A large overlap can also create excess key frames that require additional processing. Keyframes can be selected based on a spatial displacement. Keyframes can also be selected based on a sequential shifting can be selected. This type of sequential shift could, for example, mean that every tenth frame as a keyframe is selected. In one aspect, keyframes can be used when data is collected, based on any number. suitable criteria can be selected. In another aspect, keyframe pairs can be subsequently selected by examining all possible Candidate keyframes can be selected. All possible keyframe pairs can be examined and candidates can be chosen. For example, an area will be removed where there is insufficient overlap to form an attachment. More generally, any area can be selected. A suitable technique can be usefully employed to process key frames within a data set subset of a sentence. to select in order to reduce computational complexity.

[0061] Once key frames have been selected, additional processing can be carried out. For example, Full image data (e.g., full-resolution center and side channel images) for each keyframe, along with image signature data, Point cloud centroid calculations and any other measured or calculated data are stored for use to support the key frame in a three-dimensional reconstruction process as described herein.

[0062] As shown in step 814, candidate attachments can be detected. In general, an attachment is a relationship between two separate three-dimensional measurements from two different camera poses. Once an attachment is made, a Rotation and translation are determined for the path of a camera between the two poses. In a complementary way. The three-dimensional measurements from the poses can be combined to form a section of a three-dimensional model. Candidate attachments around each keyframe are analyzed, such as from the keyframe to some or all dataframes between the keyframe and the dataframes. Keyframes and adjacent keyframes. In another aspect, a candidate can attach to any other keyframe, or to reduce computational complexity, to surround each key frame within a spatial or sequential neighborhood with a Key frames can be used. Additions can be based on the originally depicted frames. It can also be useful to deform or warp two-dimensional images during registration and other steps in an attachment process, in order to to improve accuracy and / or speed. Additions can also be applied to other observed epipolar features, either additionally or instead. Relationships are based on the source data.

[0063] As shown in step 816, attachments for the complete camera path can be selected from the entire set of candidate attachments. The selection of the appendages can be based, for example, on the lowest calculated error in the resulting sections. of the three-dimensional model. In general, each key frame can be connected to one or more other key frames. can be appended, and each non-keyframe can be appended to at least one sequentially adjacent keyframe.

[0064] As shown in step 818, a graphical analysis can be performed using the key frames and the associated attachment, to calculate a global path for the camera used to obtain a three-dimensional model. The graphical analysis Each keyframe can be viewed as a node or vertex, and each attachment as an edge between a pair of accounts. A keyframe is selected as a starting point. A breadth-first search or depth-first search can be performed using the graphical representation. This will be carried out to identify attachments that can connect the current keyframe to another keyframe. Each key frame can be marked when the graphical representation is processed. A check can be performed to... to determine whether all keyframes within the graphical representation have been reached. If not all keyframes have been reached by the Once the attachments have been traversed in the graphical analysis, the largest graphical sub-representation is identified. This graphical Partial representations can be examined to determine whether the entire three-dimensional image can be modeled.

[0065] It may be that certain graphical sub-representations are not required to complete the three-dimensional representation. If the camera lingered over a particular area of ​​an object's surface, or when the camera repeatedly focused on an area If a loop is executed, the graphical subrepresentation(s) may not be needed. If a separate graphical subrepresentation is detected, To complete the three-dimensional image, an optional jump back to step 812 can be performed. For example, a set of keyframes may have been selected that does not sufficiently link from one keyframe to the next. It has key frames. By choosing a different set of key frames, a sufficient attachment can be obtained to create a to obtain a complete graphical representation of all necessary aspects of the three-dimensional image. A key framework that is too meager, which This means that it has insufficient attachments to help in constructing a graphical representation; it may indicate that another sentence Key frames should be selected. Based on the graphical analysis, a global path can be selected, and the graphical The representation can then be analyzed to optimize the path calculation.

[0066] As shown in step 820, a numerical optimization to reduce errors in the calculated camera path can be based on The available data for the complete camera path are used, such as cross-connections, remote measurements. temporarily relate them to each other. In general, the goal of numerical optimization is to calculate an error based on to minimize an error function for the camera path and / or the reconstructed three-dimensional model. A useful formulation of the The error minimization problem for a global camera path is presented below.

[0067] There may be a set of candidate camera poses, each of which includes a rotation and a translation (or position) with respect to a world coordinate system. It may also include a set of measured frame-to-frame measurements. Camera movements, each involving a rotation and will be. Let there be an example set of three keyframes with one origin “O” and three other points “A”, “B” and “C” are considered, with each of the points having a position in a three-dimensional space. In addition to the position of these points, a camera at each of these points may have a different orientation. Therefore, between Each of these points represents a translation, which means a change in position, and a rotation, which means a change in orientation. Translation and rotation values ​​comprise the motion parameter. The relationship between a point X, designated XO in the world coordinate system, and its position is defined as a coordinate system. is expressed, and the same point XA, expressed in the A-coordinate system, can be expressed as: XA = ROAXO + TOA [Eq. 3]

[0068] ROA is the rotation that transforms points from the world coordinate system to the A-coordinate system. TOA is the translation of the world coordinate system to the A- Coordinate system. It should be understood that the symbols X and T can represent a vector rather than a scalar, e.g., where X includes x, y, and z coordinate values. Furthermore, it should be understood that the symbol R can represent a matrix. The following Equations can accordingly represent the transformation between the world coordinate system and the B or C coordinate system: XB = ROBXO + TOB[Eq. 4] XC = ROCXO + TOC[Eq. 5]

[0069] By rearranging, equations 4 and 5 can be represented as follows: XO = R–1OA(XA – TOA) = R–1OB(XB – TOB)[Eq. 6]

[0070] The representation of a point in one coordinate system of the camera can be compared with the same point in another coordinate system in Relationships can be established. For example, as in equations 3–5, the coordinates of a point X can be derived from the A-coordinate system to the A-coordinate system. The B-coordinate system can be transformed as follows: XB = RABXA + TAB [Eq. 7]

[0071] The rotation RAB rotates points from the A- to the B- coordinate system and TAB translates the origin of the A- coordinate system to the B- Coordinate system.

[0072] During optimization, the pose of each camera can be optimized based on measured transformations between poses. That is, A number of camera-to-world rotations and camera-to-world translations (RON and TO) can be performed. In general, One of these can be defined as the identical rotation and zero translation, while the remaining values ​​can be optimized as described below. can.

[0073] The rotations and translations can be measured for many camera pairs. For the i-th frame-to-frame measured in this way, Let one of the cameras in the pair be camera A and the other camera B. This can also be considered the i-th addition. Let Ri AB be the measured rotation that transforms points in the A-system to the B-system, and Ti AB the coordinates of the points in the B-system. A-position expressed as in equation 7.

[0074] The rotations and translations for all cameras ROn and TOn can be optimized. Ri C,OA and Ri C,OB can be used as the Candidate rotations are defined; Ti C,OA and Ti C,OB can be defined as the candidate translations that the A and B cameras of the i- tenth addition. Furthermore, RiC,AB = RiC,OB(RiC,OA)–1 can be considered the candidate rotation from A to B and TiC,AB = TiC,OB – RiC,ABTiC,OA are defined as the candidate translation for the transformation from A to B.

[0075] Note that with sufficient additions the motion constraints become an overdetermined system of It is possible to form equations constraining the motion. If these equations are used as a starting point, a numerical solution can be developed. Optimization of the rotational and translational components of each camera is performed based on the measured attachments.

[0076] In decoupled optimization, the rotation and translation components can be optimized independently. If a Given a candidate set of camera rotations Ri C, the corresponding candidate camera-to-camera rotations Ri C,AB can be determined. The values ​​corresponding to each of the measured camera-to-camera rotations Ri AB are calculated. Consequently, the corresponding residual rotations are given by Riresidual,AB = RiC,AB(RiAB)–1. A scalar value rotation cost function can be calculated, which is given by the Candidate camera rotations depend

[0077] In equation 8, logSO(3)(R) returns the axis angle vector √, which corresponds to the rotation R. In other words, logSO(3)(R) returns the vector &ngr; returned, which has a cross product matrix [&ngr;]x, that is, the matrix logarithm of R.

[0078] Next, a similar scalar value cost function for the translation can be calculated, which depends on the candidate rotations and - depends on translations.

[0079] Equation 8 can be minimized as a nonlinear optimization; Equation 9 can be minimized as a linear optimization.

[0080] In a conventional decoupled approach to solving these simultaneous systems of equations, the rotational error function can be can be transformed into a quaternion expression to convert the numerical problem into a linear system of equations for its solution. translate. While this approach can increase computational efficiency, it offers an incomplete optimization solution.

[0081] The decoupled approach described above does not provide a truly optimal solution in the sense of maximum probability, since it does not Information from the translation section of the attachments can be used when determining the rotation. To achieve coupled optimization, Can a weighting be used to balance the contributions of the rotation and translation components to a combined cost function? To bring about balance:

[0082] Several approaches can be used to optimize this cost function, however, in one embodiment the Weights can be expressed as matrices. Different attachments can be based on a number of factors that determine the number of points. in the attachment (e.g., the shared content), the quality of a particular three-dimensional measurement, and / or any other Factors that influence the known reliability of an attachment are given different weights. In one approach, The weight matrices also take into account an anisotropic error in the collected individual points, as a result of the recording of Depth information from disparity measurements, resulting in a measurement accuracy that changes with the distance from the camera.

[0083] In some cases, equation 10 can be reformulated so that the rotational and translational weights for each attachment are decoupled. are (i.e., Wi C is a block diagonal). In particular, this can occur in the case where the motion attachments are three-dimensional. Point correspondences with an isotropic point error can be recovered. In this case, for a given attachment i, between the Camera A and camera B find the optimal solution to bring the point cloud, as seen by camera A, into agreement with the one that seen from camera B. If XiA and XiB are the positions of the center of the point cloud in the A and B systems respectively, then r can be expressed in equation 10. The residual displacement between the point cloud centers, based on the candidate camera pose, can be replaced as follows: rit,ctr = XiB – (RiC,ABXiA + TiC,AB)[Eq. 11]

[0084] Equation 10 can then be reformulated as:

[0085] This coupled optimization problem can still be considered nonlinear. It should be understood that other Further optimizations are also possible and fall within the scope of this disclosure.

[0086] In general, by minimizing equation 10, both rotational errors and translational errors can be minimized simultaneously. The weight matrices can be, for example, according to the "First Order Error Propagation of the Procrustes Method for 3D Attitude". “Estimation” by Leo Dorst, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No. 2, Feb. 2005, pp. 221–9 selected will be, which in its entirety is incorporated by reference. Once a more consistent set of motion parameters has been generated, The three-dimensional model can be updated.

[0087] In one aspect, the residual error can be used as a calibration metric. If the total error or a proportion of the error Once the error has been minimized, the residual error can be evaluated. If a minimized error falls below a certain threshold, then a Calibration of the image scanner and associated hardware is recommended based on the conclusion that the inability to obtain results with The inability to produce better quality images may be due to incorrect calibration or other malfunction of the camera system. The threshold can be empirically determined based on the specific image scanner hardware configuration, or it can be determined through experience over time for a given The system will be experienced. If a system is new or has recently been realigned, expected minimized error values ​​can be obtained. If the minimized error values ​​deviate from these expected values, a calibration status assessment marker can be set, or Another alarm or message may be generated indicating that the tool should be calibrated.

[0088] As shown in step 822, oversampling can be performed to create a three-dimensional model using data from non- To improve key frames. For example, non-key frames can be registered with nearby key frames to improve small, to create local reconstruction patches that contain the full image detail available from non-keyframes. In this way, a Path optimization is performed on a keyframe-based dataset, consequently reducing the data that constitutes a Processing is required, while additional data points from non-keyframes are needed for use in the final three-dimensional model. be stored.

[0089] It will be recognized that each of the above systems and / or methods can be implemented in hardware, software, or any combination thereof can be realized, which is suitable for the data acquisition and modeling technologies described herein. This includes the Implementation in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, together with internal and / or external memory. This can also or instead one or more application-specific integrated circuits, programmable gate arrays, programmable array logic components or any other device or devices that can be configured to process electronic signals. It is further detected. that an implementation can have a computer-executable code that is written using a structured programming language such as C, an object-oriented programming language such as C++, or any other higher or lower-level programming language (including assembly languages, hardware description languages, and database programming languages ​​and technologies) are generated and stored, can be compiled or interpreted to be executed on any of the above devices, as well as heterogeneous combinations of Processors, processor architectures, or combinations of different hardware and software. Consequently, one aspect of this is a A computer program product that includes computer executable code which, when installed on one or more computer devices, is executed, performing any and / or all of the steps described above. Simultaneously, processing can be carried out via devices such as a camera and / or a computer and / or a manufacturing facility and / or a dental laboratory and / or a server in a number of ways The functionality can be distributed, or it can be integrated into a dedicated, stand-alone device. It is intended that all such permutations and combinations fall within the scope of the present disclosure.

[0090] While the invention has been disclosed in connection with the preferred embodiments, which are shown and described in detail Various modifications and improvements will readily occur to experts. Consequently, the spirit and framework of the present invention to be limited by the preceding examples, but is to be understood in the broadest sense that is legally permissible. Summary

[0091] A three-dimensional measurement is refined by taking two-dimensional images of an object from offset camera positions. according to a three-dimensional model of the object, and any resulting discrepancies are applied, to refine the three-dimensional model, or to refine one of a number of three-dimensional measurements used to generate the three-dimensional models are used. QUOTES INCLUDED IN THE DESCRIPTION

[0092] This list of documents cited by the applicant was generated automatically and is solely for the better information of the Readers' contributions were included. The list is not part of the German patent or utility model application. The DPMA assumes no liability whatsoever. for any errors or omissions. Cited patent literature

[0093] - US 7372642 [0028, 0039] Cited non-patent literature

[0094] - “First Order Error Propagation of the Procrustes Method for 3D Attitude Estimation” by Leo Dorst, IEEE Transactions on Pattern Analysis and Machine Intelligence, B. 27, No. 2, Feb. 2005, pp. 221–9

[0086]

Claims

[1] Method for refining a three-dimensional model comprising the steps: Providing a three-dimensional model of an object; Obtaining a first two-dimensional image of the object from a first camera pose; Obtaining a second two-dimensional image of the object from a second camera pose, where the second two-dimensional image has a has a common section of a surface of the object with the first two-dimensional image; Deformation of the first two-dimensional image based on a spatial relationship between the first camera pose, the second camera pose and the three-dimensional model, in order to obtain an expected image from the second camera pose based on the first camera pose; Comparing the second two-dimensional image with the expected image to identify one or more discrepancies; and Correcting the three-dimensional model based on one or more discrepancies. [2] The method of claim 1, wherein the first camera pose and the second camera pose define a position and orientation of a single camera exhibit in two dependent positions. [3] Method according to claim 2, wherein the first camera pose and the second camera pose define a position and orientation of two offset Channels of a camera with multiple apertures. [4] The method of claim 1, wherein the first camera pose and the second camera pose define a position and orientation of a single camera exhibit in two independent positions. [5] Method according to claim 4, wherein a relationship between the first camera pose and the second camera pose is based on a The three-dimensional measurement of the object's surface is calculated from the first camera pose and the second camera pose, respectively. [6] The method of claim 1, further comprising deriving the three-dimensional model from several three-dimensional measurements of the surface of the object from several camera poses including the first camera pose and the second camera pose. [7] The method of claim 1, further comprising applying one or more discrepancies to directly generate the three-dimensional model refine. [8] The method of claim 1, further comprising the application of one or more discrepancies to obtain a three-dimensional measurement from to refine the first camera pose and / or the second camera pose in order to provide a more refined measurement. [9] The method of claim 8, further comprising refining a camera path calculation for a camera path used to generate the three-dimensional model is used, with the refined measurement being used to provide a refined camera path. [10] The method of claim 9, further comprising the use of the refined camera path and the refined measurement to achieve the to refine the three-dimensional model. [11] Method according to claim 1, wherein the three-dimensional model comprises a point cloud or a polygonal grid. [12] Method according to claim 1, wherein the object has a human dentition. [13] Method according to claim 1, wherein the second camera pose corresponds to a central channel of a camera system with multiple apertures, wherein the The central channel provides a conventional two-dimensional image of the object. [14] The method of claim 14, further comprising obtaining a third two-dimensional image of the object from a third camera pose, which a second side channel of the system with multiple apertures, and the deformation of the third two-dimensional image to an expected The image for the central channel is used for further refining the three-dimensional measurement from the multi-aperture camera system. [15] Computer program product for refining a three-dimensional model of an object, which contains a computer-executable code exhibits, which is contained on a computer-readable medium, which, when executed on one or more computer devices, performs the following steps: Providing a three-dimensional model of an object; Obtaining a first two-dimensional image of the object from a first camera pose; Obtaining a second two-dimensional image of the object from a second camera pose, where the second two-dimensional image has a has a common section of a surface of the object with the first two-dimensional image; Deformation of the first two-dimensional image based on a spatial relationship between the first camera pose, the second camera pose, and the three-dimensional model to obtain an expected image from the second camera pose based on the first camera pose; Comparing the second two-dimensional image with the expected image to identify one or more discrepancies; and Correcting the three-dimensional model based on one or more discrepancies. [16] Computer program product according to claim 15, wherein the first camera pose and the second camera pose define a position and an orientation a single camera in two dependent positions. [17] Computer program product according to claim 16, wherein the first camera pose and the second camera pose define a position and an orientation exhibiting two offset channels of a camera with multiple apertures. [18] Computer program product according to claim 15, wherein the first camera pose and the second camera pose define a position and an orientation featuring a single camera in two independent positions. [19] Computer program product according to claim 18, wherein a relationship between the first camera pose and the second camera pose is based calculated on a three-dimensional measurement of the object's surface from the first camera pose and the second camera pose. becomes. [20] Computer program product according to claim 15, further comprising code for performing the step of deriving the three-dimensional model from several three-dimensional measurements of the object's surface from several camera poses, including the first camera pose and the second camera pose. [21] Computer program product according to claim 15, further comprising code for performing the step of applying one or more It reveals discrepancies in order to directly refine the three-dimensional model. [22] Computer program product according to claim 15, further comprising code for performing the step of applying one or more exhibits discrepancies in order to refine a three-dimensional measurement from the first camera pose and / or the second camera pose, in order to obtain a to provide a more refined measurement. [23] Computer program product according to claim 22, further comprising code for performing the step of refining a Camera path calculation for a camera path that is used to generate the three-dimensional model, wherein the refined The measurement is used to provide a refined camera path. [24] Computer program product according to claim 23, further comprising code for performing the step of using the refined camera path and refined measurement to refine the three-dimensional model. [25] Computer program product according to claim 15, wherein the three-dimensional model comprises a point cloud or a polygonal grid. [26] Computer program product according to claim 15, wherein the object has a human dentition. [27] Computer program product according to claim 15, wherein the second camera pose is a central channel of a camera system with multiple apertures corresponds to the central channel being a conventional two-dimensional provides an image of the object. [28] Computer program product according to claim 15, further comprising code for performing the steps of obtaining a third two-dimensional image of the object from a third camera pose, which is a second side channel of the system with multiple apertures corresponds to, and the deformation of the third two-dimensional image into an expected image for the central channel for use in further Refining the three-dimensional measurement of the camera system with multiple apertures.

Images