Image signatures for use in motion-based three-dimensional reconstruction

Abstract

A family of one-dimensional image signatures is obtained, each representing a sequence of images in multiple translational and rotational orientations. By computing these image signatures during image acquisition, a new, current view can be rapidly compared to previous views in a manner that is less dependent on the relative orientation between a target and a search image. This and other techniques can be used in three-dimensional reconstruction processing to generate a list of candidate images, from which a full three-dimensional registration can be performed as a test for a suitable three-dimensional match. From another perspective, this procedure can be augmented by a Fourier-based method that can be selectively applied to a subset of the previous images.By switching between spatial signatures for one set of previous views and spatial frequency signatures for another set of previous views, a pattern matching system can be implemented that enables faster linking to a three-dimensional model in various practical applications.

Description

[0001] The present application claims priority over the preliminary US patent application No. filed on January 4, 2008. 61 / 019159, to which reference is made herein in its entirety. Scope of the invention

[0002] The present invention relates generally to three-dimensional images or three-dimensional imaging and in particular to image signatures. for improving alignment or fit in a motion-based three-dimensional reconstruction. Background of the invention

[0003] According to a technique for three-dimensional image reconstruction, several images or sets of images of an object are created by a The camera recorded footage moving along a path across the surface of the object. If the camera path is interrupted, either due to an intentional interruption by a user or because it is not possible to reconcile new incoming data with an existing one To connect a three-dimensional model, it may be desirable to perform a scanning process for reconstruction by reconnecting to to continue the existing camera path. Although a general three-dimensional registration is conceptually possible to obtain a current view from a camera with one or more image data frames in an existing camera path used for a reconstruction in relation to This approach becomes impractical when the three-dimensional model is formed from hundreds or thousands of image data frames, which Contains millions of surfaces or three-dimensional points.

[0004] There is therefore a need for improved techniques for finding a good match for a current camera view with one of several existing image data frames. Brief description of the invention

[0005] A family of one-dimensional image signatures is provided to represent each individual image in a sequence in multiple translational and to represent rotational orientations. By calculating these image signatures during image capture, a new, current View in a way that is less dependent on the relative alignment between a target image and a search image, quickly They can be compared with previous views. These and other techniques can be used in a three-dimensional reconstruction process. to generate a list of candidate images, among which a full three-dimensional registration will be used as a test for a suitable one Three-dimensional matching can be performed. According to another aspect, this procedure can be achieved using a Fourier-based method. to be supplemented, which is selectively applied to a subset of the earlier images. By switching between spatial signatures for a set Previous views and spatial frequency signatures for another set of previous views can be used in various practical applications. A pattern matching system will be implemented that can be linked more quickly to a three-dimensional model.

[0006] According to one aspect, a method for generating a signature for image matching described herein comprises the following steps: Providing an image containing multiple pixels; generating an average for each of several pixel rows in a central region of the Image to generate a linear matrix of series means, which is stored as a first signature; rotation of the mid-range with respect to of the image to provide a rotated center image; generating an average for each of several pixel rows in the rotated center image to to generate a linear matrix of rotated series means, which is stored as a second signature; translating the mean range with respect to of the image to provide a translated center image; generating an average for each of several pixel rows in the translated center image, to generate a linear matrix of translated series means, which is stored as a third signature; and determining an element-wise Series mean for each signature of the image that has at least the first signature, the second signature and the third signature, and storage of the element-wise series mean as a summary image signature describing the image.

[0007] The image can be a compressed version of a source image with a larger pixel count. The method can perform the translation. of the central region to several offset positions with respect to the image and the determination of another linear matrix of series means from the center region for each of the multiple offset positions. The method can involve rotating the center region into several offset orientations with respect to the image and determining another linear matrix of series means from the center region for each of the exhibit several offset orientations. The method can involve receiving a second image; generating an average for each of several pixel rows in a central region of the second image to generate a linear matrix of row means, which serves as a search signature is stored; and comparing the summari The image signature must match the search signature to identify a possible match.

[0008] According to another aspect, a computer program product described herein for generating a signature for image matching a computer-readable medium, executable code which, when installed on one or more computers, The computer system executes the following steps: providing an image containing multiple pixels; generating an average for each pixel. of several pixel rows in a central region of the image to generate a linear matrix of row means, which serves as a first signature is saved; rotate the center area relative to the image to provide a rotated center image; generate an average for Each of several pixel rows in the rotated center image is used to generate a linear matrix of rotated row averages, which serve as a second signature. is stored; translate the mid-range with respect to the image to provide a translated mid-image; generate an average value for each of several pixel rows in the translated center image, to generate a linear matrix of translated row means, which serves as a third Signature is stored; and determining an element-wise series mean for each signature of the image that is at least the first signature, which has the second signature and the third signature, and storing the element-wise series mean as a summary representing the image Image signature.

[0009] According to another aspect, a method described herein for using image signatures for image matching The steps in a three-dimensional reconstruction process are: generating an image signature for each of several images that are in a three-dimensional reconstruction is used, with each image signature having a first signature and several alignment signatures, whereby Each of the orientation signatures is calculated in the same way as the first signature, with the image undergoing at least one offset rotation- and corresponds to an offset translation position, and wherein each image signature contains a summary image signature, which is considered an average of the The first signature and each of the alignment signatures are calculated; determining a second signature for a search image that is three-dimensional. Reconstruction is to be added, with the second signature being calculated in the same way as the first signature; selecting multiple Candidate images from the multiple images based on a comparison between the second signature of the search image and the summary Image signature of each of the multiple images; selection of multiple candidate registrations from the candidate images based on a comparison between the second signature and the image signature, as well as the multiple alignment signatures for each of the candidate images; sequential Trial registration of a three-dimensional data set assigned to the search image with one of the candidate images assigned to each three-dimensional data set until a resulting registration has a residual error that is smaller than a predetermined threshold; and adding the search image to the multiple images, which involves adding the three-dimensional data set associated with the search image to the three-dimensional reconstruction is included.

[0010] Each image signature can contain a spatial frequency range representation or spatial frequency domain representation of the image, and the second The image signature can display a spatial frequency range representation or spatial frequency domain representation of the search image. Each image signature can be based on a downclocked version of several images. Each image signature can be based on a central area of ​​one of the several images. Each One of the multiple images can be a key frame in a camera path, which is used to determine the three-dimensional reconstruction. is used. The procedure can discard the search image if none of the resulting registrations has a residual error that is smaller than the specified threshold, and a new search image is retrieved. The procedure can scale the search image. exhibit such that the three-dimensional data set assigned to the search image and the data set assigned to at least one of the several images three-dimensional datasets have an essentially similar center of gravity distance.

[0011] According to another aspect, a computer program product described herein for using image signatures for a Image matching in a three-dimensional reconstruction process involves a representation stored in a computer-readable medium and displayed on a computer. executable code which, when run on one or more computer systems, performs the following steps: generating an image signature for each of several images used in a three-dimensional reconstruction, where each image signature has a first signature and has multiple alignment signatures, each of which is calculated in the same way as the first signature, wherein the image corresponds to at least one offset rotational and one offset translational position, and wherein each image signature represents a summary Image signature calculated as the average of the first signature and each of the alignment signatures; determining a second signature for a search image to be added to the three-dimensional reconstruction, where the second signature is calculated in the same way as the first signature; Select more rerer candidate images from the multiple images based on a comparison between the second signature of the search image and the summary Image signature of each of the multiple images; selection of multiple candidate registrations from the candidate images based on a comparison between the second signature and the image signature, as well as the multiple alignment signatures for each of the candidate images; sequential Trial registration of a three-dimensional data set assigned to the search image with one of the candidate images assigned to each three-dimensional data set until a resulting registration has a residual error that is smaller than a predetermined threshold; and adding the search image to the multiple images, which involves adding the three-dimensional data set associated with the search image to the three-dimensional reconstruction is included.

[0012] According to another aspect, the method described herein for using image signatures for image matching includes the steps on: Generating at least one spatial signature or at least one spatial frequency signature for each of several images; checking a first Search image regarding a match with a first subset of the multiple images based on a spatial signature for the first Search image; and checking a second search image for a match with a second subset of the multiple images based on the Location frequency signature for the second search image.

[0013] The first clause may differ from the second clause or be unique compared to the second clause. Checking the The second search image may involve sequentially checking the second search image if no suitable match was found during the check of the first search image. A match is found. The multiple images may contain images that are used in a motion-based three-dimensional reconstruction. can be used. The first subset can contain several key images that define a camera path in a motion-based three-dimensional reconstruction can be used. The first subset can contain all key images for a three-dimensional scan. For each keyframe, multiple spatial signatures can be calculated, which define the keyframe for multiple offset rotations and Represent translation positions. The second clause can contain one or more immediately preceding images in a sequence of images. which are obtained during a motion-based three-dimensional reconstruction. The first search image and the second search image can These are sequential current views obtained by a three-dimensional camera. The method can involve checking the second search image for a match with the second subset of multiple images based on a spatial signature for the second image. The search image must be displayed. The procedure can involve the alternating repetition of a check based on a spatial signature and one based on a spatial frequency signature-based verification for each new current view obtained from a three-dimensional camera, until a A match is found according to a predefined criterion. The procedure can use the match for registration. exhibit a three-dimensional reconstruction for a current view of a three-dimensional model that is composed of three-dimensional Data is obtained that is associated with the multiple images. The process may involve discarding each new, current view until the A match is found.

[0014] According to another aspect, a computer program product described herein for using image signatures for a Image matching involves a computer-readable, executable code stored on a computer, which, if executed on a or is executed on multiple computer systems, performing the following steps: generating at least one spatial signature or at least one Spatial frequency signature for each of several images; checking a first search image for a match with a first subset of the several images based on a spatial signature for the first search image; and checking a second search image for a match. with a second subset of the multiple images based on the spatial frequency signature for the second search image. Brief description of the drawings

[0015] The invention and the following detailed description of specific embodiments of the invention are described below with reference to the The following figures illustrate this.

[0016] Fig. 1 shows a three-dimensional scanning system.

[0017] Fig. 2 shows a schematic diagram of an optical system for a three-dimensional camera.

[0018] Fig. 3 shows a processing pipeline for obtaining three-dimensional data from a video camera.

[0019] Fig. 4 shows a sequence of images taken by a moving camera.

[0020] Fig. 5 shows a series of image data frames.

[0021] Fig. 6 shows an image signature for a two-dimensional image.

[0022] Fig. 7 shows an image signature with a rotational offset.

[0023] Fig. 8 shows an image signature with a translation offset.

[0024] Fig. 9 shows a window for a spatial frequency signature.

[0025] Fig. 10 shows a processing method for using image signatures for relinking with an existing three-dimensional scanning. Detailed description of the invention

[0026] In the following text, references to singular elements should be understood to include plural elements and Conversely, unless explicitly stated otherwise or evident from the text, grammatical conjunctions should express each and all of them. Disjunctive and conjunctive combinations of linked sentence parts, sentences, words, etc. are expressed, unless otherwise expressly stated. or is evident from the context.

[0027] In the following description, specific scanning techniques are described in detail, with particular attention to dental While the applications of three-dimensional imaging are addressed, it should be noted that the methods and systems described herein are not intended for use in the production of three-dimensional images. generally applicable in any environment where a search image can be arranged in several different target images, especially when the search image has an unknown three-dimensional position and orientation relative to the target image(s). All these Variations, adaptations and combinations that are obvious to those skilled in the art shall be within the scope of protection of the present invention. It should be included.

[0028] In the following description, the term “image” generally refers to a two-dimensional set of pixels, that form a two-dimensional view of an object in an image plane. The term "image set" generally refers to a A set of interrelated two-dimensional images that can be resolved into three-dimensional data. The expression "Point cloud" generally refers to a three-dimensional set of points that form a three-dimensional view of the object. which is reconstructed from several two-dimensional images. In a three-dimensional image acquisition system, several such point clouds can be generated. also registered and combined into a total point cloud constructed from images taken by a moving camera Therefore, it is clear that pixels generally represent two-dimensional data and points generally denote three-dimensional data. unless another meaning is specifically stated or is clear from the context.

[0029] The terms “three-dimensional model”, “three-dimensional surface representation”, used herein “Digital surface representation”, “three-dimensional surface map”, etc. are intended to represent any three-dimensional surface. The term "reconstruction of an object" refers to, for example, a point cloud of surface data or a set of two-dimensional polygons. or any other data representing all or part of the surface of an object, for example by capturing and / or processing three-dimensional scan data can be obtained, unless a different meaning is expressly stated or from as is clear from the context. A “three-dimensional representation” can be any of the three-dimensional representations described above. Surface representations, as well as volumetric or other representations, unless a different meaning is expressly stated. is or is evident from the context.

[0030] In general, the terms “rendering” or “rendering” refer to a two-dimensional visualization. a three-dimensional object, e.g., for display on a monitor. However, it should be noted that different three-dimensional Rendering techniques are known and suitable for use with the systems and methods described herein. For example, the ones described herein The described systems and methods are suitable for a holographic display, an autostereoscopic display, an anaglyph display, a head-mounted display Use a mounted stereo display or any other two-dimensional and / or three-dimensional representation. Therefore, rendering should The following terms should be interpreted broadly in the present description, unless a narrower meaning is explicitly stated or can be inferred from the text. context emerges.

[0031] The term “dental object” as described herein is intended to refer in the broadest sense to objects or items that are connected with These can be related to dentistry. They can be intraoral structures, such as a dentition, and in particular a human dentition, e.g. individual teeth, quadrants, full arches, pairs of arches (which may be separate or arranged in different occlusal positions), soft tissues or Soft tissue, etc., as well as bones and all other supporting or surrounding structures are included. The one used herein The term “intraoral structures” refers to both natural structures within a mouth as described above, and artificial structures, such as any dental objects described below and located within Dental objects can be arranged in the mouth. Examples of dental objects are "restorations," which are generally considered components. These are understood to be those that restore the structure or function of the existing dentition, such as crowns, bridges, Veneers, fillings or inlays, onlays, amalgams, composites and various substructures, such as crown caps, as well as temporary restorations used during the fabrication of a permanent restoration. Dental objects can also be "Prosthesis" refers to a partial or total denture that replaces the teeth with removable or permanent structures, such as a dental prosthesis or Artificial dentures, partial dentures, implants, retained prostheses, etc. Dental objects can also be “apparatuses” that used to correct, align or otherwise temporarily or permanently adjust the bit, such as removable orthodontic appliances, surgical vascular prostheses or stents, bruxism appliances, anti-snoring devices or Snoring guards, devices for indirect clasp placement, etc. Dental objects can also be “small parts”. include those that are attached to the teeth for an extended period of time, such as implant holders, implant anchors, Orthodontic braces and other orthodontic components. Dental objects can also be “interim components”. for dental manufacturing, such as dental models (full and / or partial models), wax models, lost-wax molds, etc. Trays, bases, molds, and other components used in the fabrication of restorations, prostheses, etc. Dental objects They can also be incorporated into natural dental objects, such as teeth, bones, and other intraoral structures described above, or Artificial dental objects are categorized, such as restorations, prostheses, appliances, small parts and interim components for the dental manufacturing, as described above.

[0032] Terms such as “digital dental model”, “digital dental impression”, etc. are intended to refer to three-dimensional Representations of dental objects are used in various aspects of acquisition, analysis, prescription and manufacturing. They can be used unless a different meaning is explicitly stated or evident from the context. Expressions such as... “Dental model” or “dental impression” are intended to refer to a physical model, e.g., a cast, printed, or A physically manufactured copy of a dental object. Unless explicitly stated otherwise, the term may be interpreted in various ways. The term "model" is used independently to refer to both a physical and a digital model.

[0033] It is also pointed out that expressions such as “tool” or “control”, insofar as They are used to describe aspects of a user interface and generally refer to various techniques that are used in Connections to a graphical user interface or other user interface can be used to receive user input. which triggers or controls processing, such as drop-down lists, wireless key presses, cursor and / or mouse movements (a selection) (by points, selection by range, drag-and-drop operations, etc.), checkboxes, command lines, text input fields, messages and warnings, status or progress bars, etc. A tool or control can also be any physical hardware component. be that which is related to user input, such as a mouse, a keyboard, a display, a keypad, a trackball and / or any other device that receives physical input from a user and converts the physical input into an input device for Use in a computer-controlled system is converted. Therefore, in the following description, the terms "tool" and “Control element”, etc., should be understood broadly unless a more specific meaning is given or can be inferred from the context. emerges.

[0034] Fig. 1 shows a three-dimensional scanning system that can be used with the systems and methods described herein. The system 100 can generally have a camera 102 that captures images of a surface 106 of an object 104, e.g., a dental patient, and forwards the images to a computer 108, which may have a display 110 and one or more user input devices 112, 114, e.g. a mouse 112 or a keyboard 114. The camera 102 may also have an integrated input or output device 116, e.g. a A control input element (e.g., a button, a touchpad, a thumbwheel) or a display (e.g., an LCD or LED display) to provide status information to provide.

[0035] The camera 102 can be any camera or camera system suitable for capturing images from which A three-dimensional point cloud or other three-dimensional data can be recovered. For example, camera 102 can... Use the multi-aperture system described in US Patent No. 7372642 by Rohely et al., to which it is incorporated herein by reference in its entirety. referenced Although Rohály describes a multi-aperture system, it is pointed out that similarly any multi-aperture system can be used. usable, which is suitable for reconstructing a three-dimensional point cloud from a number of two-dimensional images. According to a In its multi-aperture configuration, the camera 102 can have several apertures, including a central aperture located along the optical central axis of a lens. is positioned, forming a central channel for camera 102, in combination with any associated imaging hardware.

[0036] In such embodiments, the central channel can provide a conventional video image of the scanned object, while several Axially offset channels provide image sets containing disparity information that is used for three-dimensional reconstruction of a surface. can be. In other embodiments, a separate video camera and / or a separate channel may be provided to capture the same information. The result is to achieve a video of an object that corresponds temporally to a three-dimensional scan of the object, preferably from the same perspective or from a perspective with a fixed, known relationship to the perspective of camera 102. Camera 102 can in addition, or instead, feature a stereoscopic, triscopic or other multiple camera or other configuration, at which involves holding several cameras or optical paths in a fixed relationship to each other in order to create two-dimensional images of an object. to obtain several different perspectives. The camera 102 can perform suitable processing to derive a three-dimensional image. A point cloud can be derived from one or more image sets, or each two-dimensional image set can be sent to an external processor. can be transmitted, for example, the computer 108 described later. In other embodiments, the camera 102 can transmit structured data. Use light, laser scanning, tachymetric distance measurement, or any other technique suitable for capturing three-dimensional data. or is suitable for two-dimensional data that can be resolved into three-dimensional data. Although the techniques described later To be able to use video data appropriately that is captured by a video-based three-dimensional scanning system, it is clear that any other A three-dimensional scanning system can be supplemented with a video acquisition system that simultaneously or Alternatively, the data is captured synchronously with the acquisition of three-dimensional data.

[0037] According to one embodiment, the camera 102 is a freely positionable handheld probe with at least one user input device 116 , e.g. a button, a lever, a dial, a thumbwheel, a switch, etc. for controlling the image capture system 100 by a user, e.g., to start and stop scanning processes. According to one embodiment, the camera 102 can be used for tooth scanning. be shaped and dimensioned. In particular, the camera 102 can be shaped and dimensioned for intraoral scanning and data acquisition, e.g. B. by inserting it into the mouth of a patient to be imaged and guiding it over an intraoral surface 106 at a suitable distance to to capture surface data of teeth, gums, etc. Through such a continuous data acquisition method, the camera 102 can... Capture a point cloud of surface data with sufficient spatial resolution and accuracy to generate dental objects, e.g., prosthetics, from it. Small parts, apparatus, etc., can be manufactured either directly or via various intermediate processing steps. In other embodiments Surface data from a dental model, e.g., a dental prosthesis, can be captured to ensure a suitable fit using a previous scan of the corresponding dentition, e.g., a tooth surface prepared for the prosthesis.

[0038] Although not shown in Fig. 1, it is clear that several supplementary lighting systems can be used appropriately during image acquisition. are. For example, the ambient lighting can be enhanced by one or more spotlights illuminating object 104 in order to the to accelerate image acquisition and improve depth of field (or depth of spatial resolution). The Camera 102 can also or instead, a stroboscopic light, a flash light or another light source to supplement the illumination of object 104 during image capture exhibit.

[0039] The object 104 can be any object, a group of objects, a part of an object or any other item. In particular, object 104, with regard to the dental techniques discussed herein, can be a human set of teeth taken from the mouth of a The patient is photographed intraorally. A three-dimensional image can be created through a scanning process, depending on the specific purpose of the scan. A representation of part or all of the dentition can be captured. Therefore, a digital model of a tooth, a Tooth quadrants or a complete group of teeth including two opposing arches, as well as soft tissue or any other Relevant intraoral structures are captured. Multiple images can be captured through scanning, e.g., a tooth surface before and after preparation for a restoration. As mentioned below, this data can be used for subsequent modeling, e.g., for... Designing a restoration or determining a boundary line there for use. During the scanning process, a center channel of camera 102 or a separate video system can record a video of the The teeth are captured from the viewpoint of camera 102. In other embodiments, for example, if a finished product is being processed, the teeth can be captured from the viewpoint of camera 102. The surface preparation is fitted on a trial basis, the scanning reveals a dental prosthesis, such as an inlay or a filling, a A crown or any other dental prosthesis, small dental parts, a dental appliance, etc. Item 104 can also be, or instead of, a Dental model, such as a plaster cast, a wax model, an impression or a negative impression of a tooth, teeth, or soft tissue. or any combination thereof.

[0040] The computer 108 can, for example, be a personal computer or another processing device. According to one embodiment The Computer 108 is a personal computer with a 2.8 GHz dual-Opteron central processing unit (CPU), 2 gigabytes of RAM, and a TYAN Thunder K8WE. The system consists of a motherboard and a 250-gigabyte hard drive with a rotational speed of 10,000 revolutions per minute. According to one embodiment, the system can be operated in this manner. that more than 5000 points per image set are captured in real time using the techniques described herein and an accumulated A point cloud consisting of several million points can be captured. Naturally, this point cloud can be further processed to create a subsequent... To enable data handling, for example by decimating the point cloud data or by creating a corresponding grid of surface data. is generated. The term "real-time" used here generally means that there is no time lag between processing and display. A perceptible delay occurs. In a video-based scanning system, real time refers in particular to processing within the Time between images / frames of video data, which according to specific video techniques is between approximately fifteen images / frames per second and The frame rate can fluctuate by approximately thirty images / frames per second. In particular, the processing power of the computer 108 can vary depending on its size. of object 104, the speed of image acquisition, and the desired spatial resolution of three-dimensional points vary. Computer 108 can also have peripheral devices, such as a keyboard 114, a display 110 and a mouse 112, which are connected to a To enable users to interact with the camera system 100. The display 110 can be a touchscreen display suitable for this purpose. User input is received through direct physical interaction with the Display 110. According to another aspect, the display can It should be an autostereoscopic display or a similar display suitable for displaying stereo images.

[0041] Communications between the computer 108 and the camera 102 can be made via any suitable communication link, for example via a wired connection or a wireless connection based on, for example, the IEEE 802.11 standard (also (known as wireless Ethernet), Bluetooth or any other suitable wireless standard using a radio frequency, an infrared or other wireless communication medium. In medical imaging or other sensitive applications, Applications can secure wireless image transmission from camera 102 to computer 108. Computer 108 can Generate control signals and transmit them to camera 102, which, in addition to image acquisition commands, contain conventional camera control commands. can, for example, focus and zoom control commands.

[0042] In an example of the general operation of a three-dimensional image acquisition system 100, the camera 102 can capture two-dimensional images. Capture image sets at a video rate while the camera is moved across the surface of an object. The two-dimensional image sets can be forwarded to computer 108 to derive three-dimensional point clouds. The three-dimensional data for each new The captured two-dimensional image set can be derived using various techniques and adapted to the existing three-dimensional data. Data can be adapted or "linked" to it. Such a system can use camera motion estimation to to avoid the need for independent tracking of the position of camera 102. A useful example of such a technique. is described in US patent application no. 11 / 270135, filed on November 9, 2005, to which reference is made herein in its entirety. Reference is made to this. However, it should be noted that this example is not intended as a limitation, and that the information contained herein The described principles are applicable to a wide range of three-dimensional image acquisition systems.

[0043] The display 110 can be any display suitable for playback at a video rate or other rate with a level of detail is suitable, one that corresponds to the captured data. Suitable displays include cathode ray tube displays, liquid crystal displays, LED displays, etc. Display 110 can generally be operationally connected to the computer 108 and is suitable for receiving display signals from the computer. The display can be a CRT display or a flat screen, a three-dimensional display (e.g., an anaglyph display), an autostereoscopic display A three-dimensional display or any other two-dimensional or three-dimensional playback device. In some cases, Depending on the guidance methods, the display can have a touchscreen interface, which may include, for example, capacitive or resistive touchscreen technologies or Touchscreen techniques using surface acoustic waves (also known as dispersive signals) or any Another technique was used to detect a physical interaction with the screen 110.

[0044] The system 100 can comprise a medium that is usable on a computer or computer-readable. The medium that is usable on a computer usable medium 118 can contain one or more memory chips (or other chips, such as a processor, which have a memory) exhibiting), optical disks, magnetic disks or other magnetic media, etc. The medium usable on a computer 118 can in various embodiments be a removable storage device (e.g. a USB device, a tape drive, an external hard drive, etc.), a remote storage device (e.g., storage connected in a network), volatile or non-volatile computer memory, etc. exhibit. The medium 118, which can be used on a computer, can contain computer-readable commands that are executed by the computer 108. are intended to be used to carry out the various processes and procedures described herein. The computer-usable medium 118 It can also or instead store data received from camera 102, a three-dimensional model of object 104, computer code for rendering, processing and display, etc.

[0045] Fig. 2 shows an optical system 200 for a three-dimensional camera, which in conjunction with the systems described herein and The method can be used, e.g., for the camera 102 described above with reference to Fig. 1.

[0046] The optical system 200 can have a primary optical device 202, which is of any type of image processing system usable. In general, a primary optical device herein refers to an optical system with a optical channel. Typically, this optical channel utilizes at least one lens and has a shared image plane within the optical system, although modifications to this may be expressly described in the following description or otherwise indicated. the context will be clear. The optical system 200 can be a single main lens, a group of lenses, an object lens, mirror systems (e.g. B. conventional mirrors, digital mirror systems, digital light processors, etc.) confocal mirrors and other optical devices, which are suitable for use with the systems described herein. The optical system 200, for example, can be used in a stereoscopic or other multi-image camera systems are used. Other optical devices include, for example, holographic optical systems. Elements or similar. In various configurations, the primary optical device 202 may have one or more lenses, such as for example, an object lens (or a group of lenses) 202b, a field lens 202d, a relay lens 202f, etc. The object lens 202b can The field lens 202d may be arranged at or near the entrance pupil 202a of the optical system 200. The relay lens 202f is arranged in the first image plane 202c of the optical system 200. It can focus bundles of light rays in the optical system 200. amplify. The optical system 200 can also incorporate components such as aperture elements 208 with one or more apertures 212 , a focusing adjustment device 210 with one or more focusing adjustment elements 204, one or more scanning devices 218 and / or several sensors 214a, 214b, 214c.

[0047] The optical system 200 can be designed for active wavefront scanning, which is to be understood as any technique, which is used to sample a sequence or group of optical data from an object 220 or objects, including optical data that to help probe two-dimensional or three-dimensional properties of the object, using optical data to detect movement, for speed measurement or object tracking, etc. Further details of an optical system, known as the optical system System 200 of Fig. 2 is usable and is described in US Patent No. 7372642, to which reference is made in its entirety herein. In general, it can be seen that, although Fig. 2 represents one embodiment of an optical system 200, various modifications are possible. are.

[0048] Fig. 3 shows a three-dimensional reconstruction system 300, which includes a high-speed processing pipeline and a A high-precision processing pipeline is used. Generally, 330 three-dimensional objects are to be processed via the high-speed processing pipeline. Data is provided in real time, such as with a video frame rate or video speed determined by an assigned The display is used, while the high-precision processing pipeline 350 achieves the highest precision according to camera measurements. to be provided that are subject to any external computing processing or time constraints imposed by System hardware or an intended use of the results may be imposed. A data source 310, such as the one above The described camera 102 transmits 300 images to the system. data or similar data. Data source 310 can, for example, be hardware components, e.g., LED ring lights, rod sensors, a Frame grabber, a computer, an operating system and any other hardware and / or software components for obtaining data exhibiting features used for three-dimensional reconstruction. Images from data source 310, e.g., center-channel images, which are conventional Video images are included, and side-channel images containing disparity data that are used to recover depth information. can be fed to the real-time processing controller 316. The real-time processing controller 316 can also process camera control information. or provide other feedback information for data source 310, which may be used for a subsequent data retrieval or to specify Data is used that has already been obtained in data source 310 and that the real-time processing controller 316 requires. Images with Full resolution images and related image data can be stored in a full-resolution image memory 322. The stored Images can, for example, be fed to the high-precision processing controller 324 during processing, or can be viewed during processing. The following processing steps are used for an image view by a user.

[0049] The real-time processing controller 316 can supply the high-speed (video rate) processing pipeline 330 images or frames for the real-time reconstruction of three-dimensional surfaces from two-dimensional source data. In an exemplary case In this embodiment, two-dimensional images from an image set, such as side-channel images, can be replaced by a two-dimensional Image registration module 332 can be registered. Based on the results of the two-dimensional image registration, a module 334 can be registered. To create a three-dimensional point cloud, generate a three-dimensional point cloud or another three-dimensional representation. The three-dimensional point clouds of individual image sets can be combined for three-dimensional linking using a module 336. Finally, the linked measurements can be combined into an integrated system by a module 338 to generate a three-dimensional model. three-dimensional models can be combined. The resulting model can be saved as a three-dimensional high-speed model 340.

[0050] The high-precision processing controller 324 can feed images or frames to the high-precision processing pipeline 350. For separate image sets, two-dimensional image registration can be performed by a module 352 for two-dimensional image registration. Based on the results of the two-dimensional image registration, a module 354 can be used to generate a three-dimensional image. A point cloud can be a three-dimensional point cloud or another three-dimensional representation. Three-dimensional point clouds The individual image sets can be linked using a module 356 for three-dimensional linking. A global Motion optimization, also referred to here as global path optimization or global camera path optimization, can be achieved through a module 357 for global motion optimization (GMO module) to reduce errors in the obtained three-dimensional model 358. In general, the camera's path while capturing the image frames can be calculated as part of the three-dimensional reconstruction process. In a post-processing refinement procedure, the calculation of the camera path can be optimized – i.e., the accumulation. Errors along the length of the camera path can be corrected by additional frame-to-frame motion estimation by part or all of the camera. Global orbit information is minimized. Based on global information, such as individual data frames in the image memory 322 , the three-dimensional high-speed model 340 and the intermediate results in the high-precision processing pipeline 350 , The high-precision model 370 can be processed to reduce errors in the camera path and errors in the reconstructed model. As Further refinement can be achieved by projecting a grid onto the high-speed model using a 360° grid projection module. The resulting Images can be rotated or warped using a rotation module 362. Rotated images can be used to adjust the orientation and the To facilitate the linking of images, for example by reducing the initial error in motion estimation. The rotated images can be fed to module 352 for two-dimensional registration. The feedback of the three-dimensional high-precision model The 370 command in the pipeline can be repeated until some metric is obtained, for example, a link accuracy or a minimum. Error threshold.

[0051] Fig. 4 shows a coordinate system for three-dimensional measurements using a system such as the one described above. described optical system 200. The following description is intended to provide useful context and should not be construed as restrictive. Senses are understood. In general, an object 408 within an image plane 402 of a camera has the world coordinates {Xw, Yw, Zw} in a World coordinate system 410 , camera coordinates {Xc, Yc, Zc} in a camera coordinate system 406 and image set coordinates {xi, yi, di (xi, yi)} for i = 1 up to N points or pixels within a processing grid of the field of view 402 , where di is a disparity vector 412 be draws, which contains one or more disparity values ​​that define the z-axis offset (Zc) or depth 404 of a point in the image plane 402 based on an x-axis and / or y-axis shift in the image plane 402 between several physically offset apertures or other imaging channels. The processing grid can be any overlay or raster for an image or Other two-dimensional data can be understood to identify the positions where processing will take place. Although a A processing grid can be a regular grid of positions in a square, rectangular, triangular, or other pattern. The processing grid may also or instead exhibit irregular patterns that are random or according to the specific processing The object is selected. The disparity vector 412 can, for example, be used to shift a potentially existing object. The center channel is displayed for the camera. In general, the disparity vector 412 encodes the depth, and in various other In three-dimensional imaging systems, this disparity vector 412 can be replaced by one or more other measured quantities. will be used to encode the depth. Therefore, expressions such as disparity vector, disparity value, and disparity data, and similar terms, should generally be defined as follows: It is understood that they include one or more arbitrary scalar and / or vector quantities that can be measured by a system for detecting Depth information can be measured. Furthermore, the term used herein can refer to three-dimensional measurement or measured values ​​in general. denotes any form of data that encodes three-dimensional data, such as groups of three-dimensional images, of from which disparity vectors can be obtained, the disparity field (of disparity vectors) itself, or a disparity field derived from the disparity field Three-dimensional surface reconstruction. In image-based three-dimensional reconstruction, a camera model can be used to to relate disparity vectors within a camera's field of view to depth. The camera model can then be based on a optical modeling or another physical basis, theoretically or empirically through observation, or through a combination of these. Techniques can be determined, and it can be calibrated to compensate for optical aberrations, lens defects, and any other physical changes or To correct characteristics of a particular physical system.

[0052] Although a single image plane 402 is shown for illustrative purposes, it can be seen that a multi-aperture camera (or another A multi-channel system can have several physically offset optical channels, each providing a different image plane, whereby the Differences in feature positions (xy offset) between the images for each optical channel can be represented as the disparity field. In various specific processing steps, the disparity data can use a single image plane as a reference plane, e.g., the Center channel image plane of the camera.

[0053] Fig. 5 shows a sequence of image data frames. As described above, each of the image data frames can contain 500 image sets, such as two-dimensional images from a central channel and from one or more side channels of a three-dimensional camera. In one embodiment, the side channels can exhibit disparities relative to each other or to the central channel, encoding depth information. which are used to recover points in three dimensions, although the methods and systems described herein are suitable various other techniques for capturing three-dimensional frame information for motion-based three-dimensional Reconstruction can be adapted. Frames 500 can contain some keyframes 502 and several other, non-keyframes 508. exhibit. For selecting keyframes and storing various types of more complete data with 508 different frames. Several techniques are possible for 502 key images. According to one aspect, the 502 key images can be selected in such a way that... They contain sufficiently overlapping data to reconstruct a camera path using only keyframes. Keyframes 502 can also be based on, or instead on, sequential frame separation, physical proximity, or a Another suitable metric will be selected to generate a subset of the full frame sequence 500 for improved processing.

[0054] Processing limitations or design preferences can also influence the key image selection. For example, the The total number of keyframes may be limited, or the number of frames between keyframes may be limited. According to another For example, a minimum and / or maximum movement range between camera positions (and / or orientations) may be desired for key images. According to one aspect, image sets for frame 508 can be discarded between the key images, so that only the three-dimensional ones remain. Data and camera translation / rotation for each non-keyframe 508 remain. Simultaneously, more complete data can be used for keyframes 502. stored, such as full-resolution image sets, sequence numbers, links to other keyframes 502, below described image signature data (e.g., compressed images, image signatures with rotation and translation offset, etc.), etc. During a three-dimensional scanning can be A camera path is created, with each frame having 500 shared overlapping elements with each preceding frame and each exhibits the following framework. If this scanning process is either intentional by a user or due to reconstruction errors. is interrupted (i.e., an object moves outside the camera's scanning volume; excessive offset results in a new A frame cannot be joined to a preceding frame, or any event results in a loss of image data or the (Camera path), can perform processing to relink a new frame 520 for a current camera view with one created by the System-stored any other frame 500 can be activated. Techniques for generating image signatures for use in this Processing is described with reference to Figures 6 to 9. A process for using these image signatures is described with reference to... Fig. 10 described.

[0055] According to one aspect, images used for a signature, such as image 500 in Fig. 5, can be linked to a substantially The usual scale or center of gravity distance can be adjusted to normalize magnification for fitting purposes. This can, for example, by determining a center of gravity of the three-dimensional reconstruction recovered from a data frame and estimating or Calculating a depth or distance from the camera position for this single point can be implemented. By scaling various Target images (key images and / or other frames) and / or search images (e.g., of the current camera view) can be aligned to a common depth. Magnification effects are reduced.

[0056] Fig. 6 shows an image signature for a two-dimensional image. An image 600, which for example is an image of any of the above The described image data frame can contain multiple pixels that encode two-dimensional image data. For example, image 600 can... a full-resolution image or a compressed image, e.g., a half-resolution image, a quarter-resolution image, or any image with any other size. It should be noted that the techniques described herein are suitable for downscaled images. or lower-resolution images can be used to save processing resources. For example, a full-resolution image can The image, measuring 1024×768 pixels, is converted into an image measuring 64×48 pixels and used as image 600, with regard to which image signature calculations are performed. To obtain an image signature, the central area 602 of image 600, which contains several pixels 604, can be identified. It was pointed out that, although a specific number of pixels are shown in Fig. 6, this number of pixels is only for illustrative purposes. It serves its purpose and should not restrict the scope of protection of the invention. Any suitable number of pixels can be used, e.g., square pixels. Matrices, such as 8×8, 16×16, 32×32 matrices, or any other suitable square or rectangular pixel window of the Image 600 or a pixel window of any other shape and size. An average can be calculated from each set of pixel values ​​in the center area. to be formed to provide a series mean 606 for this series, and the series means 606 can be stored in a linear matrix 608 a signature for image 600 is stored. In one embodiment, this image signature is calculated for each current image. when attempting to establish a link to a frame sequence after an interruption, as described below.

[0057] Fig. 7 shows an image signature with a rotational offset. The image 700 can be rotated (or the central area 702 can be (can be rotated), and series of pixel values ​​in the mid-range 702 can be averaged to obtain a series mean, and the Series means for the middle range 702 can be stored in a linear matrix 704, which has an image signature with a rotation offset. 706 represents. Any number of rotational offset signatures can be generated. For example, image 700 can be generated in steps of, e.g., Rotated 10 degrees over a full circle or over a segment of a circle, e.g. from -40 degrees to +40 degrees, around an original orientation will be (resulting in nine rotational offset image signatures). It should be noted that any rotation or twisting mentioned herein This denotes a relative rotation of image 700 with respect to the central region 702. From an analytical perspective, it should not matter whether image 700 or The center area is rotated 702, although from a computational point of view it may be more efficient to use one of these options. For example If the center area 702 is rotated, fewer calculation steps may be required to obtain values ​​within the rotated area. to determine the window of the central area 702. In this context, a rotation, in particular the relative rotation of these images without Referring to which of the two images is rotated into the coordinate system of the other. In embodiments of the above The described handheld camera can adjust rotation increments during a scan operation to accommodate expected manual alignments of a scan operation. be centered.

[0058] Fig. 8 shows an image signature with a translation effect. The image 800 can be translated (or the central area 802 can be (translated), and series of pixel values ​​in the middle. The values ​​of the 802 range can be averaged to obtain a series mean, and the series means for the middle range of 802 can be compiled in a a linear matrix 804 is stored, representing an image signature with a translation offset of 806. Any number of Translation offset signatures can be generated. For example, the image can be 800 in steps of, say, one pixel in the x- and y-axes, or along translated along a single axis (where a rotation may contain translation information along the orthogonal axis). It is to recognize that a translation mentioned herein denotes a relative translation of image 800 with respect to the central region 802. Analytically From a purely technical standpoint, it shouldn't matter whether the image is translated using 800 or the central area using 802, although it can be computationally efficient. to use one of these options. For example, if the middle range is translated as 802, a smaller number of calculation steps may be required. This may be necessary to determine values ​​within the translation window of the middle range 802. In this context, a translation is required. in particular, the relative translation of these images without reference to which of the two images is placed in the coordinate system of the other. is translated.

[0059] According to one embodiment, nine rotations can be used, with nine translations being provided for each rotation. This results in 81 image signatures, covering different orientations of an image for each key image of data. If rotations and translations are centered around the original image orientation, one of the image signatures may be a zero rotation-zero translation signature. included for the original image orientation. To improve processing speed when searching for a current image in the image data catalog. To improve this, these multiple image signatures can be averaged on an element-wise basis to obtain a single linear matrix that The middle signature represents an image. It can be seen that, although certain motion-based systems have multiple signatures for each data frame. two-dimensional images, a single image from each image set, e.g. a conventional still image of a center channel or a A similar camera can be used to match signatures for a search image and the target images in the above. to improve the described framework catalog 500.

[0060] Fig. 9 shows a window for a spatial frequency signature. In addition to the rotations and translations of an image described above, a spatial frequency signature for an image 900 using a window 902 to select pixels 904 in the image 900 and to execute a two-dimensional transformation, e.g. a fast Fourier transform (“FFT”), can be obtained to obtain the windowed to arrange pixel values ​​in a spatial frequency domain representation. As described in more detail below, this spatial frequency signature can be used in This can be used in combination with the spatial signatures described above to search for images that correspond to a current view. to agree, to improve in the framework data catalog 500 for an existing scan.

[0061] Fig. 10 shows a processing method for using image signatures for relinking with an existing three-dimensional Sampling. Processing 1000 can begin by receiving a live frame, as shown in step 1002. This live frame (the (also referred to as the current view) represents a current image data frame from a three-dimensional camera, which, for example, is any The camera described above can be from a current position (and orientation) of the camera. In one embodiment, the live frame is an image set with a center channel image containing a conventional two-dimensional image of a scanned object, included together with two side channel images from offset optical axes.

[0062] As shown in step 1004, the live frame can be linked to a preceding frame (which contains a keyframe). or may be a non-key image), to recover a camera translation or rotation for the live frame and recovered to add three-dimensional data to a three-dimensional model. If the linking is successful, so that, for example, the When three-dimensional data from successive data frames are registered with sufficient accuracy, the processing can begin. Progress from step 1000 to step 1006. If the link is unsuccessful, processing from step 1000 can optionally proceed to step 1008 or 1010. progress, as discussed generally below.

[0063] As shown in step 1006, if a current or live frame has been linked to the existing data frame catalog It must be determined whether the frame is a key image. This determination can be based on any of the criteria discussed above, such as... for example, a relative overlap with other key images, a sequential separation from a preceding key image, a spatial separation (of the recovered camera position) from other key images, etc.

[0064] If the frame is not a keyframe, Processing can proceed from step 1000 to step 1012, where the non-key image is stored. This step can, for example, Saving recovered data, e.g., camera position, camera orientation, a three-dimensional point cloud, etc., and discarding it. of source data, such as the full-resolution image set for the frame. In one embodiment, the full-resolution Image sets for each frame after the most recent keyframe or for an immediately preceding non-keyframe can be temporarily stored. In such an embodiment, when a new keyframe is generated, the non-keyframes between the new keyframe and the keyframe can be stored. The key image and the preceding key image may be deleted in whole or in part.

[0065] If the frame evaluated in step 1006 is selected as the key image, the frame can be added to the sample data catalog as A keyframe can be added. Besides saving the full-resolution data for the keyframe (in step 1012), Additional processing can be performed on keyframes. For example, any of the above can be applied to each keyframe. The described signatures are calculated. In one embodiment, this includes several linear matrices for multiple offset rotational and Translation positions, as generally described above. This can also provide a mean value for these linear matrices for use in Signature-based searches are included for customizing frame content.

[0066] In one embodiment, a key image (or a reduced version of a key image) can be processed to obtain an average value. to obtain for each of several pixel rows in a central region of the image and to provide a linear matrix of row means, which can be used as A first signature is stored. The central area can then be rotated relative to the image by any number of permutations and are translated, whereby for each permutation a linear matrix of rotated and / or translated series means is calculated. The obtained Matrices can be stored as image signatures for the key image. Furthermore, an element-wise average of these matrices can be calculated. and are stored as a summary image signature for the image. After the key image has been processed, the processing can be extended to 1000. Step 1012 continues, where key image data is stored, and processing 1000 can then jump back to step 1002, where a The next live data frame will be received from a camera.

[0067] If, after returning to step 1002, a link is lost for any reason (either due to a camera or a If processing fails (due to operator error or a specific user command), it can optionally proceed to step 1008, where a spatial signature is used for image matching, or to step 1010, where a spatial frequency signature is used for image matching. These methods can be based, for example, on whether the live frame has an even or odd sequence frame number, or used alternately using another suitable weighting technique or a technique without weighting.

[0068] As shown in step 1008, a spatial signature can be used to extract two-dimensional data from the live frame with to compare keyframes or other frames stored in the frame catalog. In one embodiment, a spatial signature is used. calculated for the live setting using the technique described above with reference to Fig. 6. Although for this signature a Since any translation or rotation can be used, the signature can be advantageously positioned with respect to a centered, unrotated window for a A compressed version of the center channel image from the live frame is applied.

[0069] Various techniques can be used to utilize the spatial signature information for image matching. The procedure can be implemented as follows to refine the search area over several steps before a full three-dimensional search is performed. An attempt is being made to register recovered three-dimensional data. The live frame signature can be based on the summary image signature. (a single linear matrix of the element-wise mean of signatures of different orientations for the frame, as above (as described), which is calculated for each of the key images, are compared with key images in a framework catalog. The comparison can be, for example, as a normalized cross-correlation of the summary image signature and the live frame signature, or using a any other suitable similarity measure can be calculated. The computationally quite simple comparison can, for example, for all key images in the catalog for a scan or all key images for a specific area of ​​interest in the reconstructed three-dimensional model. The key image comparisons obtained based on the summary image signature can be found under Using any suitable method for identifying multiple candidate images, e.g., the n best candidate images on a quantitative basis, or all key images with a consistent rating that is above a predetermined threshold, sorted or evaluated become.

[0070] For each of the candidate images, a comparison can be made between the image signature for the live frame and each rotated or translated version. Signature is executed for each (key image) candidate image. In an exemplary embodiment, in which for each key image 81 Signatures can be used; 81 comparisons can be performed for each of the candidate images. The resulting comparisons can be... These images can then be ordered or evaluated using any suitable similarity measure to identify key images that good candidates for registration. These can be an absolute number (e.g., the key images with the five best individual results, or the five best key images based on individual results) or represent a variable number based on a predetermined threshold. based.

[0071] As shown in step 1016, these registration candidates – e.g. the key images that contain five (or fewer or more) show the best individual matches with the live image signature – in a full linking operation, such as The operation described above with reference to step 1004 is registered with the live frame on a trial basis. For each link A quality assessment can be based, for example, on the error in the link or another residual or effort function for the link. can be determined. In one embodiment, the first link can be with a key image that represents a predefined threshold quality criterion. Once fulfilled, the next key image for the catalog can be selected. In another embodiment, a link can be established with each The key image must have full resolution, and the best link should be selected based on a quality assessment. If at least... If a link to a registration candidate is successful, the live frame can be added to the catalog as a key image, as shown. as shown in step 1014. If no link to a registration candidate is successful, processing can proceed from step 1000 to step 1002. jump back to where a new live frame is captured.

[0072] As shown in step 1010, instead of (or in certain embodiments in addition to) a spatial signature, A spatial frequency signature can be used to compare a live frame with other data frames. This can generally be done a frequency range comparison of the live frame with other frames using, for example, a windowed fast Fourier transform (FFT) of a compressed image or any other frequency domain representation. In a In this embodiment, the spatial frequency signature can be compared with one or more current non-key images after the last key image. In another embodiment, the spatial frequency signature can be compared exclusively with the single most recent data frame. will be, which has been successfully linked to the existing three-dimensional model. It is clear that in this context a comparison is necessary. estimated rotation and translation based on spatial frequency spectra may include, with a linkage based on these parameters. An attempt is being made. It is clear that in other cases, e.g., when an attempt is made to perform a scan with a sensor oriented differently in the direction of rotation, To continue scanning, rotation information from the spatial frequency signature of a live frame and a registration candidate will be used. This movement is expected, for example, when a handheld camera, e.g., the camera described with reference to Fig. 1, is operated by a hand. when a user switches to the other hand and a user attempts to continue scanning at the same physical location where The sampling process was interrupted. In these cases, the spatial frequency signature can be appropriately applied to current key images, when suitable matching camera positions can be found.

[0073] In one embodiment, the processing can be carried out on a regular basis (e.g., even and odd numbers). frame) between a comparison using a spatial signature (step 1008) and a comparison using a Switch spatial frequency signature (step 1010). This alternating comparison allows for the use of a spatial signature. The processed frames can be distinguished from the frames processed using a spatial frequency signature, e.g., under Using a spatial signature for all frames in one iteration and using a spatial frequency signature for one or more Current sequential non-key images in a next iteration. Generally, it should be apparent that various other protocols are suitable. can be used to switch between these two methods in order to obtain combined benefits of each matching technique, while the The computational load is reduced. In other implementations, both techniques can be used simultaneously for each new live data frame. or processing 1000 can use certain combinations of these procedures. As shown in step 1016, the results can If a trial registration is successful, it will be added to the scan catalog as keyframe 1014. If registration is not attempted or is unsuccessful, processing may jump back to step 1002, where the next step begins. A live frame is received from the camera.

[0074] It is understandable that any of the systems and / or methods described above may be implemented in hardware and / or software can be implemented that is suitable for the data acquisition and modeling techniques described herein. This includes an implementation using one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital Signal processors or other programmable devices in conjunction with internal and / or external memory. Additionally or Instead, one or more application-specific integrated circuits, programmable gate arrays, programmable array- Logic components or one or more other devices may be provided that can be configured to transmit electronic signals to process. It is also understandable that an implementation may consist of code executable on a computer, which runs under Using a structured programming language, such as C, or an object-oriented programming language, such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages) and database programming languages ​​and techniques) is generated, which can be stored and compiled or interpreted so that it can be used on the aforementioned devices, as well as heterogeneous combinations of processors, processor architectures or combinations various hardware and software. Therefore, according to one aspect, a computer program product is provided that runs on a computer. contains executable code which, when executed on one or more computer systems, can perform any of the above-described The process can be distributed across multiple facilities in several ways. will be, for example, a camera and / or a computer and / or a manufacturing facility and / or a dental laboratory and / or a server, or the The entire functionality can be integrated into a dedicated, standalone device. All such permutations and combinations are intended to within the scope of protection of the present invention.

[0075] Although the invention has been explained in connection with the illustrated and described in detail preferred embodiments As such, various modifications and improvements are apparent to those skilled in the art. Therefore, the present invention is not to be limited by the foregoing. The examples shown are not limited, but are to be understood in the broadest sense permissible under the law. Summary: Image signatures for use in motion-based three-dimensional reconstruction

[0076] A family of one-dimensional image signatures is obtained, each of which is derived from a sequence of images in several translational and Represent rotational orientations. By calculating these image signatures during image acquisition, a new, up-to-date view of a A way that is less dependent on a relative alignment between a target and a search image, quickly with previous views They can be compared. These and other techniques can be used in three-dimensional reconstruction processing to create a list. to generate candidate images, of which a full three-dimensional registration serves as a test for a suitable three-dimensional match. can be executed. According to another aspect, this method can be complemented by a Fourier-based method that selectively can be applied to a subset of the earlier images. By switching between spatial signatures for a set of earlier views. and spatial frequency signatures for another set of previous views, a pattern matching system can be implemented that is available in various practical applications enable faster linking to a three-dimensional model. QUOTES INCLUDED IN THE DESCRIPTION

[0077] This list of documents cited by the applicant was generated automatically and is solely for the better information of the Readers' contributions were included. The list is not part of the German patent or utility model application. The DPMA assumes no liability whatsoever. for any errors or omissions. Cited patent literature

[0078] - US 7372642 [0035, 0047]

Claims

[1] Method for generating a signature for image matching, comprising the steps: Providing an image containing multiple pixels; Generating an average for each of several pixel rows in a central region of the image to create a linear matrix of row means. generate which is stored as a first signature; Rotating the center area relative to the image to provide a rotated center image; Generating an average for each of several pixel rows in the rotated center image to create a linear matrix of rotated row averages. generate a second signature that is stored as a second signature; Translating the central area of ​​the image to provide a translated central image; Generating a mean value for each of several pixel rows in the translated center image to create a linear matrix of translated row means. generate a third signature that is stored as a third signature; and Determining an element-wise series mean for each signature of the image that includes at least the first signature, the second signature, and the third signature. exhibits a signature, and stores the element-wise series mean as a summary image signature describing the image. [2] Method according to claim 1, wherein the image is a compressed version of a source image with a larger number of pixels. [3] Method according to claim 1, further comprising the step: translating the central area to several offset positions with respect to the image and Determining another linear matrix of series means from the mid-range for each of the multiple offset positions. [4] Method according to claim 1, further comprising the step of: rotating the central area into several offset orientations with respect to the image and Determining another linear matrix of series means from the mid-range for each of the several offset orientations. [5] Method according to claim 1, further comprising the steps: Receiving a second image; Generating an average value for each of several pixel rows in a central region of the second image to provide a linear matrix of Series means, which are stored as a search signature; and Comparing the summary image signature with the search signature to identify a possible match. [6] Computer program product for generating a signature for image matching, comprising a computer-executable code that runs on is stored on a computer-readable medium, and which, when executed on one or more computer systems, performs the steps executes: Providing an image containing multiple pixels; Generating an average for each of several pixel rows in a central region of the image to create a linear matrix of row means. generate which is stored as a first signature; Rotating the center area relative to the image to provide a rotated center image; Generating an average for each of several pixel rows in the rotated center image to create a linear matrix of rotated row averages. generate a second signature that is stored as a second signature; Translating the central area of ​​the image to provide a translated central image; Generating a mean value for each of several pixel rows in the translated center image to create a linear matrix of translated row means. generate a third signature that is stored as a third signature; and Determining an element-wise series mean for each signature of the image that includes at least the first signature, the second signature, and the third signature. exhibits a signature, and stores the element-wise series mean as a summary image signature describing the image. [7] Computer program product according to claim 6, wherein the image is a compressed version of a source image with a larger number of pixels. [8] Computer program product according to claim 6, further comprising a code that includes the step of translating the middle area to several Offset positions with respect to the image and determining a further linear matrix of series means from the mid-range for each of the several executes offset positions. [9] Computer program product according to claim 6, further comprising a code that performs the step of rotating the central area into several staggered steps Orientations with respect to the image and determination of a further linear matrix of series means from the mid-range for each of the several executes offset alignments. [10] Computer program product according to claim 6, further comprising code that performs the steps: Receiving a second image; Generating an average value for each of several pixel rows in a central region of the second image to provide a linear matrix of Series means, which are stored as a search signature; and Comparing the summary image signature with the search signature to identify a possible match. [11] Method for using image signatures for image matching in three-dimensional reconstruction processing, comprising the steps: Generating an image signature for each of several images used in a three-dimensional reconstruction, where the image signature It has a first signature and several alignment signatures, each alignment signature being calculated in the same way as the first. Signature, wherein the image corresponds to an offset rotational and / or translational position, and wherein each image signature is a summary signature exhibits, which is calculated as an average of the first signature and each of the alignment signatures; Determining a second signature for a search image to be added to the three-dimensional reconstruction, where the second signature is based on calculated in the same way as the first signature; Selecting multiple candidate images from the multiple images based on a comparison between the second signature of the search image and the summary signature of each of the several images; Selecting multiple candidate registrations from the candidate images based on a comparison between the second signature and the Image signature and based on the multiple alignment signatures for each of the candidate images; sequential trial registration of a three-dimensional data set assigned to the search image with a three-dimensional data set, which is assigned to each of the candidate images until a received registration shows a residual error that is smaller than a specified value. threshold; and Adding the search image to the multiple images, which involves adding the three-dimensional data set associated with the search image to the three-dimensional reconstruction is included. [12] Method according to claim 11, wherein each image signature has a spatial frequency domain representation of the image and the second image signature has a The search image shows a spatial frequency range representation. [13] Method according to claim 11, wherein each image signature is based on a downclocked version of the multiple images. [14] Method according to claim 11, wherein each image signature is based on a central area of ​​the multiple images. [15] The method of claim 11, wherein each of the multiple images is a key image in a camera path, which is used to obtain the three-dimensional reconstruction is used. [16] Method according to claim 11, further comprising the steps: Discard the search image if none of the received registrations has a residual error smaller than the specified threshold; and Creating a new search image. [17] Method according to claim 11, further comprising the step of scaling the search image such that the three-dimensional image associated with the search image data set and the three-dimensional data set that is assigned to at least one of the several images, are substantially the same have a center of gravity distance. [18] Computer program product for using image signatures for image matching in three-dimensional reconstruction processing, with a computer-executable code stored on a computer-readable medium, and which, when placed on one or more The computer setup is executed, performing the following steps: Generating an image signature for each of several images used in a three-dimensional reconstruction, where each image signature It has a first signature and several alignment signatures, each alignment signature being calculated in the same way as the first. Signature, wherein the image corresponds to an offset rotational and / or translational position, and wherein each image signature is a summary signature exhibits, which is calculated as an average of the first signature and each of the alignment signatures; Determining a second signature for a search image to be added to the three-dimensional reconstruction, where the second signature is based on calculated in the same way as the first signature; Selecting multiple candidate images from the multiple images based on a comparison between the second signature of the search image and the summary signature of each of the multiple images; Selecting multiple candidate registrations from the candidate images based on a comparison between the second signature and the Image signature and based on the multiple alignment signatures for each of the candidate images; sequential trial registration of a three-dimensional data set assigned to the search image with a three-dimensional data set, which is assigned to each of the candidate images until a received registration shows a residual error that is smaller than a specified value. threshold; and Adding the search image to the multiple images, which involves adding the three-dimensional data set associated with the search image to the three-dimensional reconstruction is included. [19] Computer program product according to claim 18, wherein each image signature contains a spatial frequency domain representation of the image and the second The image signature displays a spatial frequency range representation of the search image. [20] Computer program product according to claim 18, wherein each image signature is based on a downclocked version of the multiple images. [21] Computer program product according to claim 18, wherein each image signature is based on a central region of the multiple images. [22] Computer program product according to claim 18, wherein each of the multiple images is a key image in a camera path, which is used to obtain the three-dimensional reconstruction is used. [23] Computer program product according to claim 18, further comprising code that performs the steps: Discard the search image if none of the received registrations has a residual error smaller than the specified threshold; and Creating a new search image. [24] Computer program product according to claim 18, further comprising code that performs the step of scaling the search image such that the three-dimensional data set assigned to the search image and the three-dimensional data set assigned to at least one of the several images, have an essentially equal distance between the centers of gravity. [25] Method for using image signatures for image matching, comprising the steps: Generating at least one spatial signature or at least one spatial frequency signature for each of several images; Checking an initial search image for a match with an initial subset of multiple images based on a spatial arrangement Signature for the first search image; and Checking a second search image for a match with a second subset of the multiple images based on a Location frequency signature for the second search image. [26] Method according to claim 25, wherein the first clause differs from the second clause. [27] Method according to claim 25, wherein checking the second search image comprises sequentially checking the second search image when during No suitable match was found when checking the first search image. [28] Method according to claim 25, wherein the multiple images include images that are in a motion-based three-dimensional reconstruction be used. [29] Method according to claim 25, wherein the first subset comprises several key frames which are used to define a camera path in a motion-based three-dimensional reconstruction can be used. [30] Method according to claim 25, wherein the first subset comprises all key images for three-dimensional scanning. [31] Method according to claim 25, wherein for each frame several spatial signatures are calculated which form the key image for several Represent offset rotational and translational positions. [32] The method of claim 25, wherein the second subset comprises one or more immediately preceding images in a sequence of images, which are obtained during a motion-based three-dimensional reconstruction. [33] The method of claim 25, wherein the first search image and the second search image are sequential current views generated by a three-dimensional camera. [34] Method according to claim 33, further comprising the step of checking the second search image for a match with the second Subset of multiple images based on a spatial signature for the second search image. [35] Method according to claim 25, further comprising the step of alternating repetition of a spatial signature-based Testing process and a spatial frequency signature-based testing process for a new current view derived from a three-dimensional The camera will be received until a match is found according to a predefined criterion. [36] Method according to claim 35, further comprising the step of using the conformity for a registration of a three-dimensional Reconstruction for a current view of a three-dimensional model obtained from three-dimensional data that each of the are assigned to multiple images. [37] The method of claim 35, further comprising the step of discarding each new current view until a match has been found. [38] Computer program product for using image signatures for image matching with a computer-executable code that is stored on a computer-readable medium, and which, when executed on one or more computer systems, performs the steps executes: Generating at least one spatial signature or at least one location frequency signature for each of several images; Checking an initial search image for a match with an initial subset of multiple images based on a spatial arrangement Signature for the first search image; and Checking a second search image for a match with a second subset of the multiple images based on a Location frequency signature for the second search image. [39] Computer program product according to claim 38, wherein the first clause differs from the second clause. [40] Computer program product according to claim 38, wherein checking the second search image is the sequential checking of the second search image indicates that no suitable match was obtained when checking the first search image. [41] Computer program product according to claim 38, wherein the multiple images contain images that are arranged in a motion-based three-dimensional Reconstruction can be used. [42] Computer program product according to claim 38, wherein the first subset comprises several keyframes which define a camera path can be used in a motion-based three-dimensional reconstruction. [43] Computer program product according to claim 38, wherein the first subset comprises all key images for three-dimensional scanning. [44] Computer program product according to claim 38, wherein for each key image several spatial signatures are calculated which the Represent key image for multiple offset rotational and translational positions. [45] Computer program product according to claim 38, wherein the second subset is one or more immediately preceding images in a sequence of Features images obtained during a motion-based three-dimensional reconstruction. [46] Computer program product according to claim 38, wherein the first search image and the second search image are sequential current views generated by a three-dimensional camera can be obtained. [47] Computer program product according to claim 46, further comprising the step of checking the second search image for a match with the second subset of multiple images based on a spatial signature for the second search image. [48] ​​Computer program product according to claim 38, further comprising a code that performs the step of alternating repetition of a process on a spatial signature-based verification process and a spatial frequency signature-based verification process for a new current view executes the image obtained from a three-dimensional camera until a match is found according to a predetermined criterion. [49] Computer program product according to claim 48, further comprising a code that performs the step to use the match for a Performs registration of a three-dimensional reconstruction for a current view of a three-dimensional model created by Three-dimensional data is obtained, which is assigned to each of the multiple images. [50] Computer program product according to claim 48, further comprising code that performs the step to discard each new current view, until a match was found.