Image processing method, scanning device and storage medium

Abstract

The application relates to an image processing method, a scanning device and a storage medium. The method is applied to a scanning device and comprises the following steps: performing global scanning on a scanned object to obtain a global point cloud, and constructing a mesh model of the scanned object by using the global point cloud; collecting a local point cloud and a first local color image of a local region of the scanned object; displaying the first local color image on the local region corresponding to the mesh model based on the registration relationship between the local point cloud and the global point cloud; and collecting a second local color image corresponding to the local region in response to a shooting instruction, wherein the resolution of the first local color image is lower than that of the second local color image. The above method can balance the image acquisition efficiency and image accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of scanning, and in particular to an image processing method, a scanning device and a storage medium. BACKGROUND

[0002] Three-dimensional scanning is a process of collecting accurate spatial geometric information (shape, size, position, etc.) of a real object or environment surface, and converting and processing it into a digital three-dimensional model that can be recognized and operated by a computer. If it is necessary to generate a texture image synchronously during scanning, a low-resolution mode is usually adopted to ensure that the scanning frame rate requirement can be met, but this will result in insufficient resolution of the final generated texture image. SUMMARY

[0003] A first aspect of the embodiments of the present application provides an image processing method applied to a scanning device, including: performing global scanning on a scanned object to obtain a global point cloud, and constructing a mesh model of the scanned object through the global point cloud; collecting a local point cloud and a first local color image of a local region of the scanned object; based on a registration relationship between the local point cloud and the global point cloud, texture information of the first local color image is displayed on the local region corresponding to the mesh model; and in response to a shooting instruction, a second local color image corresponding to the local region is collected, wherein the resolution of the first local color image is lower than that of the second local color image.

[0004] In some embodiments of the present application, after the second local color image corresponding to the local region is collected, the first local color image or the second local color image is further mapped to a local mesh of the local region.

[0005] In some embodiments of the present application, the collection of the local point cloud, the first local color image and the corresponding second local color image of the scanned object is further completed under different viewing angles.

[0006] In some embodiments of the present application, in response to the collection of all second local color images of the scanned object being completed, all the second local color images are fused to generate a global color image of the scanned object; and the global color image is mapped to the mesh model constructed by the global point cloud.

[0007] In some embodiments of the present application, the operation of fusing all the second partial color images to generate a global color image of the scanned object in response to the completion of the acquisition of all the second partial color images of the scanned object comprises: generating a first image sequence based on the acquisition sequence of the second partial color images in response to the completion of the acquisition of all the second partial color images of the scanned object; determining a local region corresponding to a second partial color image that does not meet the preset requirement in response to a quality detection result of any second partial color image in the first image sequence not meeting the preset requirement, and acquiring a replacement color image; replacing the second partial color image that does not meet the preset requirement with the replacement color image to obtain a second image sequence; and fusing the second image sequence to generate a global color image of the scanned object.

[0008] In some embodiments of the present application, the second partial color image acquired later covers the common region between the second partial color image acquired earlier and the second partial color image acquired later.

[0009] In some embodiments of the present application, the display area of the scanning device comprises a first display window, a second display window and a third display window, the first display window is configured to display the grid model, the second display window is configured to display the second partial color image, and the third display window is configured to display a second partial black-and-white image corresponding to the second partial color image, wherein the first display window is displayed full screen on the display area, the second display window and the third display window are suspended on the first display window, and the size of the second display window and the third display window is smaller than the size of the first display window.

[0010] In some embodiments of the present application, the first partial color image is mapped onto the local point cloud.

[0011] In some embodiments of the present application, the shooting instruction is triggered after the first partial color image is displayed on the local region of the grid model corresponding to the local region, and the triggering manner comprises at least one of automatic triggering, touch triggering and voice triggering.

[0012] The second aspect of the embodiments of the present application further provides a scanning device, which comprises a processor and a memory, and the processor is used to execute a computer program stored in the memory to realize the image processing method according to any one of the first aspect.

[0013] The third aspect of the embodiments of the present application further provides a scanning device, which comprises a projector configured to project light rays to a scanned object; one or more monochrome cameras configured to collect reflected light of the scanned object for the light rays; one or more color cameras configured to collect color images of the scanned object; and a processor configured to: obtain a global point cloud of the scanned object based on the reflected light, and construct a mesh model of the scanned object through the global point cloud; collect a local point cloud of a local region of the scanned object based on the reflected light, and collect a first local color image of the local region using the one or more color cameras; display the first local color image on the local region corresponding to the mesh model based on a registration relationship between the local point cloud and the global point cloud; and collect a second local color image corresponding to the local region using the one or more color cameras in response to a shooting instruction, wherein a resolution of the first local color image is lower than a resolution of the second local color image.

[0014] The fourth aspect of the embodiments of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image processing method according to any one of the first aspect.

[0015] In the image processing method provided by the embodiments of the present application, the global scanning of the scanned object obtains a global point cloud, which can not include texture information of the scanned object or only collect a low-resolution texture image, thereby improving the scanning efficiency and accuracy. The mesh model of the scanned object is constructed through the global point cloud, which converts unordered point clouds into structured surfaces and provides a geometric framework for subsequent point cloud registration. The local point cloud of the local region of the scanned object and the first local color image with low resolution are collected, the first local color image associated with the local point cloud is displayed on the local region corresponding to the mesh model through the registration relationship between the local point cloud and the global point cloud, which provides a preview of the texture effect of the local region, and ensures that the second local color image with high resolution collected under the registration can cover the local region, i.e., the second local color image can be mapped to the correct position of the scanned object. Therefore, the embodiments of the present application display the low-resolution color image on the local region using the registration of the local point cloud and the global point cloud, and indicate the collection of the second local color image with high resolution corresponding to the local region through the above registration, which improves the collection efficiency and image accuracy of the second local color image. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a structural schematic diagram of a scanning device provided by an embodiment of the present application.

[0017] Figure 2is a structural schematic diagram of a scanning device provided by another embodiment of the present application.

[0018] Figure 3 is a flowchart of an image processing method provided by an embodiment of the present application.

[0019] Figure 4 is a display interface diagram of a display screen provided by an embodiment of the present application.

[0020] Figure 5 is a flowchart of a global color image generation provided by an embodiment of the present application.

[0021] Figure 6 is a flowchart of a first local color image processing provided by an embodiment of the present application.

[0022] Figure 7 is a display interface diagram of a display screen provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] For the convenience of understanding, exemplary descriptions of some concepts related to the embodiments of the present application are given for reference.

[0024] It should be noted that “at least one” in the present application means one or more, and “multiple” means two or more than two. “And / or” describes the association between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms “first”, “second”, “third”, “fourth” and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0025] Three-dimensional scanning is a process of collecting accurate spatial geometric information (shape, size, position, etc.) of a real object or environment surface, and converting and processing it into a digital three-dimensional model that can be recognized and operated by a computer. In the scenario of three-dimensional scanning, if it is necessary to synchronously collect a texture image of the three-dimensional model during scanning, a low-resolution mode is usually adopted to ensure that the scanning frame rate requirement can be met, but this will result in insufficient resolution of the finally generated texture image, which is difficult to meet the high-precision application requirements.

[0026] In addition, if the texture image is collected again on the basis of completing the three-dimensional scanning, the mapping relationship between the three-dimensional model and the texture image needs to be calculated, which cannot ensure the accuracy of the mapping and affects the precision of the texture mapping.

[0027] Therefore, in order to solve the technical problem that the related art cannot simultaneously consider the collection efficiency and image accuracy of the texture image, the embodiment of the present application provides an image processing method, a scanning device and a storage medium, which can collect a first local color image corresponding to a local region of a scanned object, and collect a second local color image corresponding to the local region based on a registration relationship between the local point cloud and the global point cloud, so as to ensure the collection accuracy of the second local color image. The first local color image is displayed on the local region of the mesh model corresponding to the scanned object, which can present the first local color image in a visual manner, so that the user can evaluate the display effect of the second local color image through the first local color image. First, the structure of the scanning device will be described.

[0028] Figure 1 The structure of an exemplary scanning device 10 is shown. In the depicted embodiment, the scanning device 10 includes a frame structure 20 of the scanning device 10, imaging modules 30 located at the frame structure 20, and a display screen 40. The imaging modules 30 can be arranged staggered with respect to each other such that the field of view of each imaging module at least partially overlaps. In some embodiments, the imaging modules 30 can include five cameras, i.e., a first camera 31, a second camera 32, a third camera 33, a fourth camera 34, and a fifth camera 35. The imaging modules 30 can also include two groups of light projectors, i.e., a first light projector group 36 and a second light projector group 37, each of which can include one or more light projectors, for example, three light projectors. Each light projector can include a light source configured to emit a light beam to a lens, a collimating lens, and a diffractive optical element, etc. The light beam collimated by the collimating lens propagates to the diffractive optical element, which replicates the focused light beam to form a speckle pattern or a stripe pattern and projects it to the scanned object / region. The speckle image or the stripe pattern reflected by the scanned object / region back to the imaging modules 30 is collected to further obtain three-dimensional information of the scanned object / region by a processing chip (not shown in the figure). In other embodiments, the light projector can be an image projector, such as a digital micromirror device, a liquid crystal display projector, or an organic electroluminescent display projector, etc.

[0029] In some embodiments, the light projectors of the first light projector set 36 can include a single identical light source, e.g., a light source that emits infrared light, white light, blue light, or one of the other visible monochromatic lights. In other embodiments, multiple light projectors of the first light projector set 36 are all configured to emit light having a wavelength between 405 nm and 1100 nm, e.g., all configured to emit near-infrared light having a wavelength between 780 nm and 1100 nm. In other embodiments, the light projectors of the first light projector set 36 can include multiple different light sources, e.g., one or two light projectors include a light source that emits infrared light, e.g., one or two light projectors include a light source that emits near-infrared light having a wavelength between 780 nm and 1100 nm, and another light projector includes a light source that emits blue light or white light. For example, a first light projector of the first light projector set 36 includes a first light source that emits near-infrared light, a second light projector of the first light projector set 36 includes a second light source that emits white light, a third light projector of the first light projector set 36 includes a third light source that emits blue light, or a first light projector of the first light projector set 36 includes a first light source that emits near-infrared light, a second light projector of the first light projector set 36 includes a second light source that emits near-infrared light, and a third light projector of the first light projector set 36 includes a third light source that emits white light / blue light. In some examples, multiple projectors of the first light projector set 36 can be implemented as separate units, or as separate units in one area.

[0030] In some embodiments, the light projectors of the second light projector group 37, e.g. the first light projector, can comprise a single identical light source, e.g. a light source emitting one of infrared light, white light, blue light or other visible monochromatic light. In other embodiments, the light projectors of the second light projector group 37 are all configured to emit light having a wavelength between 405 nm and 1100 nm, e.g. all configured to emit blue light having a wavelength between 435 nm and 480 nm. In other embodiments, the light projectors of the second light projector group 37 can comprise a plurality of different light sources, e.g. one or two light projectors comprising a light source emitting blue light, e.g. one or two light projectors comprising a light source emitting blue light having a wavelength between 435 nm and 480 nm, and further light projectors comprising a light source emitting near infrared light or white light. For example, the first light projector of the second light projector group 37 comprises a first light source emitting blue light, the second light projector of the second light projector group 37 comprises a second light source emitting white light, the third light projector of the second light projector group 37 comprises a third light source emitting near infrared light, or the first light projector of the second light projector group 37 comprises a first light source emitting blue light, the second light projector of the second light projector group 37 comprises a second light source emitting blue light, and the third light projector of the second light projector group 37 comprises a third light source emitting white / near infrared light. In some examples, the plurality of projectors of the second light projector group 37 can be implemented as separate units, or as separate units in one area.

[0031] In some embodiments, the first light projector group 36 and the second light projector group 37 can be configured to be selectively turned on or off in adaptation to different scanning modes of the scanning device 10.

[0032] The type of camera of the first camera 31, the second camera 32, the third camera 33 and the fourth camera 34 is typically a monochrome (e.g. black and white) camera, and depends on the type of light source(s) used in the first light projector group 36 and the second light projector group 37. In some embodiments, the first camera 31, the second camera 32, the third camera 33 and the fourth camera 34 can be monochrome, visible color spectrum or near infrared cameras, and the light projectors of the first light projector group 36 are infrared light projectors or near infrared light projectors, and the light projectors of the second light projector group 37 are blue light projectors.

[0033] In some embodiments, a first camera 31 and a second camera 32 are used together as a first camera combination, and a third camera 33 and a fourth camera 34 are used together as a second camera combination. The first camera 31 and the second camera 32 are configured to have the same focal length, e.g., a closer focal length or a farther focal length, and the third camera 33 and the fourth camera 34 are configured to have the same focal length, e.g., a farther focal length or a closer focal length, and the focal lengths of the first camera combination and the second camera combination are not the same. For example, the first camera combination or the second camera combination can be selected to capture the reflections on the object from either a set of light projectors based on the adapted focal length, and the light projector set used can also be selected based on, for example, the scanning speed, scanning accuracy required by the environment, and the characteristics of the object.

[0034] In some embodiments, the first camera 31, the second camera 32, the third camera 33, and the fourth camera 34 may use any suitable shutter technology, including but not limited to rolling shutters, global shutters, mechanical shutters, and optical liquid crystal display (LCD) shutters. In some embodiments, the fifth camera 35 may be a color camera (also referred to as a texture camera), which may be configured to capture / acquire color / texture images with different resolutions, for example, to capture a first local color image and a second local color image as described herein. The texture camera may use any suitable shutter technology, including but not limited to rolling shutters, global shutters, mechanical shutters, and optical liquid crystal display (LCD) shutters. In some embodiments, the first camera 31, the second camera 32, the third camera 33, the fourth camera 34, and the fifth camera 35 may have similar configurations to improve matching confidence and speed. In some embodiments, the imaging module 30 can also use fewer projector groups, such as only one projector group and fewer cameras, such as only two cameras, to complete the acquisition of reflected light and color textures, for example, omitting the second projector group 37 (first projector group 36), for example omitting the third camera 33 (first camera 31) and the fourth camera 34 (second camera 32).

[0035] like Figure 1 As shown, the first camera 31, the second camera 32, and the first light projector group 36 can be located side-by-side on one surface of the frame structure 20, with the first camera 31, the second camera 32, and the first light projector group 36 spaced apart from each other and all facing directly forward of the surface. The third camera 33, the fourth camera 34, and the second light projector group 37 can be located side-by-side on one surface of the frame structure 20, with the third camera 33, the fourth camera 34, and the second light projector group 37 spaced apart from each other and all facing directly forward of the surface.

[0036] The data connection (such as a serial communication connection) between the scanning device 10 and one or more computer processors (not shown in the figure) can allow the transfer of data collected by the first camera 31, the second camera 32, the third camera 33, the fourth camera 34 and the fifth camera 35 so that it can be processed to derive 3D measurements of the surface of the object / object being scanned. The one or more computer processors can be implemented in a remote computing system (electronic device) or, alternatively, can be part of the scanning device 10 itself.

[0037] Exemplarily, each light projector group can comprise a single light projector or can also have two or more (e.g. three) light projectors. The light projectors can be configured to project visible or non-visible light, coherent or non-coherent light. In some embodiments, the light projectors can comprise one or more light sources consisting of lasers (e.g. Vertical Cavity Surface Emitting Lasers (VCSELs), Edge Emitting Lasers (EELs), solid-state lasers and semiconductor lasers) and / or one or more LEDs (or OLEDs).

[0038] The light projectors can be configured to project a structured light pattern consisting of a plurality of light slices arranged side by side to each other, which can appear as elongated light fringe bands when the light slices are projected onto the surface of the object. The elongated light fringe bands are non-intersecting elongated light fringe bands and, in some implementations, the elongated light fringe bands can be substantially parallel to each other. In some embodiments, the light projectors can be programmable light projector units capable of projecting more than one light pattern. For example, the light projectors can be configured to project different structured line patterns. In some embodiments, the light projectors can emit light having a wavelength between 405 nm and 1100 nm.

[0039] In some examples, the two images of the object can be captured simultaneously using the first camera 31, the second camera 32 or using the third camera 33, the fourth camera 34, an image processing can be applied to a computational method implemented, for example, by one or more processors, or to a computational method implemented, for example, by an electronic device to derive 3D measurements of the surface of the object / object being scanned.

[0040] In some examples, a texture of the object can be captured using the second camera 32 simultaneously to the capture of the object images by the first camera 31, the second camera 32 or by the third camera 33, the fourth camera 34, and the texture can be applied to a computational method implemented, for example, by one or more processors, or to a computational method implemented, for example, by an electronic device to map on the 3D measurements of the surface of the object / object being scanned.

[0041] In some examples, using a film / membrane with band-pass filter function fixed on the lens of the camera (e.g., the first camera 31, the second camera 32, the third camera 33, and the fourth camera 34) can match the wavelength of the projector(s), which can help reduce light source interference from ambient light and other projectors.

[0042] In some examples, the intrinsic and extrinsic parameters of the first camera 31, the second camera 32, the third camera 33, the fourth camera 34, and the fifth camera 35 are measured using a calibration board with a geometric distance true value measured in advance by a high-precision method such as photogrammetry. The measurement process usually includes a series of continuous measurements using a calibration camera of the scanning device 10 after adjusting the calibration board to different positions, and calculating the spatial position and orientation of the first camera 31, the second camera 32, the third camera 33, the fourth camera 34, and the fifth camera 35 by identifying the positions of the reference (marker) points / areas / lines in the calibration images, thereby completing the calibration of the intrinsic and extrinsic parameters of the first camera 31, the second camera 32, the third camera 33, the fourth camera 34, and the fifth camera 35.

[0043] In some examples, the display screen 40 is configured to display a three-dimensional model of the object captured by the imaging module 30. The display screen 40 can be arranged opposite to the imaging module 30. The display screen 40 can include, for example, a touch function or physical buttons / touch areas attached to the surrounding area of the display screen 40, which can be used to touch and move, zoom, or turn on / off the three-dimensional model of the object displayed on the display screen 40. In some examples, the display screen 40 can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini led, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), etc.

[0044] Figure 2 Another example structure of the scanning device 10 is shown. As shown, the scanning device 10 includes, but is not limited to, a projector 110, at least one monochrome camera 120, at least one color camera 130, a display screen 140, a memory 150, and a processor 160. Figure 2

[0045] ​In other embodiments of the present application, the projector 110, the at least one monochrome camera 120, the at least one color camera 130, and the display screen 140 can be devices external to the scanning device 10. The scanning device 10 can receive data sent by the at least one monochrome camera 120 and the at least one color camera 130, or the scanning device 10 can poll the at least one monochrome camera 120 and the at least one color camera 130 to obtain data collected by the at least one monochrome camera 120 and the at least one color camera 130. In addition, the scanning device 10 can also display a mesh model and the like through the external display screen 140.

[0046] The above is only an example, and the present application is not limited to the specific forms of the scanning device 10, the projector 110, the at least one monochrome camera 120, the at least one color camera 130, the display screen 140, the memory 150, and the processor 160.

[0047] The scanning device 10 can include, but is not limited to, an oral scanning device, a facial scanning device, a CT (Computed Tomography) scanning device or a CBCT (Cone beam Computer Tomography) scanning device, a professional scanner, an industrial scanner, and the like. The oral scanning device includes an intraoral scanner and an extraoral scanner. The scanning device can be a handheld scanning device or a fixed scanning device. The scanning device can realize three-dimensional reconstruction of a scanned object or scene such as teeth, a human face, a human body, an industrial product, an industrial equipment, an artifact, an artwork, a prosthesis, a medical device, a building, and the like. The present application is not limited in this regard.

[0048] The projector 110 is used to project light (such as phase-shifted fringe, random speckle, or orthogonal grid) to the scanned object, which is geometrically deformed after being projected onto the surface of the scanned object.

[0049] The monochrome camera 120 can be a black-and-white camera, which has a higher signal-to-noise ratio than the color camera 130. The monochrome camera 120 is synchronized with the projector 110 and is used to capture reflected light that is geometrically deformed after being projected onto the surface of the scanned object. The scanning device 10 converts the reflected light that is geometrically deformed into sub-pixel level phase / difference data, and solves to generate micron-level precision point cloud through a triangulation model.

[0050] The color camera 130 can also be referred to as a texture camera, which is used to collect color images (also referred to as texture images).

[0051] The display screen 140 can be used to display a mesh model constructed from point cloud data, and can also be used to display a mesh model to which a color image is pasted. In addition, it can also be used to display a scanning process of the scanned object. The present application is not limited to the content displayed by the display screen 140.

[0052] The memory 150 can include one or more Random Access Memory (RAM) and one or more Non-Volatile Memory (NVM). The Random Access Memory can be directly readable and writable by the processor 160, and can be used to store executable programs (e.g. machine instructions) of operating systems or other programs that are running, and can also be used to store data of users and applications, etc.

[0053] The Random Access Memory can include Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.

[0054] The Non-Volatile Memory can also store executable programs and store data of users and applications, etc., and can be loaded in advance into the Random Access Memory for direct reading and writing by the processor 160. The Non-Volatile Memory can include magnetic disk storage devices, Flash Memory.

[0055] The memory 150 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 160. The one or more computer programs include a plurality of instructions which, when executed by the processor 160, can implement the image processing method executed on the scanning device 10.

[0056] In other embodiments, the scanning device 10 further includes an external memory interface for connecting an external memory to achieve the expansion of the storage capacity of the scanning device 10.

[0057] Processor 160 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0058] The processor 160 provides computing and control capabilities; for example, the processor 160 is used to execute computer programs stored in the memory 150 to implement the image processing method described above.

[0059] Understandably, scanning device 10 can be used for, for example Figure 1 Or such as Figure 2 The structures shown can all be used to implement the image processing method, scanning device, and storage medium provided in the embodiments of this application.

[0060] Figure 3 This is a flowchart of an image processing method provided in one embodiment of this application, applied in a scanning device (e.g., Figure 1 or Figure 2 The scanning device 10). Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0061] Step S301: Perform a global scan on the object being scanned to obtain a global point cloud, and construct a mesh model of the object being scanned using the global point cloud.

[0062] In some embodiments of this application, the object being scanned can be any object or organism whose geometry needs to be measured, such as a car, medical device, product parts, teeth, human body model, etc. This application does not limit the type or shape of the object being scanned. To accurately detect the object being scanned, a small number of markers can be affixed to the object before scanning. These markers can be reflective.

[0063] The scanning device can perform a global scan of the object being scanned using a projector and at least one monochrome camera. A global scan means that through multi-view, blind-spot-free data acquisition and fusion technology, the three-dimensional geometry and topology of the entire surface of the scanned object can be completely reconstructed.

[0064] The scanning device can project light rays to the scanned object by using the projector, and the light rays can be geometrically deformed on the surface of the scanned object. The scanning device collects the deformed reflected light by using at least one monochrome camera. The scanning device converts the deformed reflected light into sub-pixel level phase / difference data, and calculates a point cloud with micron level precision by using a triangulation model, so as to obtain a global point cloud of the scanned object. In the case that the scanning device only uses the monochrome camera, the global point cloud does not include texture information (for example, color) of the scanned object.

[0065] In some embodiments of the present application, after the global scanning is completed, the scanning device can perform denoising and registration processing on the global point cloud, to obtain a processed point cloud. An initial mesh is obtained by using the processed point cloud for surface reconstruction, and optimization strategies such as hole repairing and smoothing denoising are performed on the initial mesh, to obtain a mesh model of the scanned object.

[0066] It should be noted that, in the global scanning, the embodiments of the present application can still acquire color / texture images of the scanned object by using a color camera or a texture camera, or the color camera / texture camera is turned off in the global scanning, and only point cloud data is acquired, which is not limited in the present application.

[0067] The above embodiments describe the processing of the global point cloud after the global scanning is completed. In addition, during the scanning process, at least part of the mesh model can be constructed based on the real-time acquired point cloud, until the mesh model is obtained after the global scanning is completed.

[0068] In step S302, a local point cloud and a first local color image of a local region of the scanned object are acquired.

[0069] In some embodiments of the present application, the different perspectives include all perspectives for completing the global scanning. The scanning device can perform local scanning on the scanned object by using the projector, the at least one monochrome camera, and the at least one color camera synchronously.

[0070] For example, the projector can project light rays to the local region of the scanned object, and the light rays can be geometrically deformed on the surface of the scanned object. The deformed reflected light is collected by using the at least one monochrome camera, so as to obtain initial point cloud data of the scanned object in a point cloud coordinate system. The surface of the scanned object is collected under visible light (non-coded light) by using the at least one color camera, and the reflected light carries original texture information, so as to obtain initial texture data of the scanned object in a texture coordinate system.

[0071] To ensure that the monochrome camera and the color camera capture the same local region, the initial point cloud data and the initial texture data can be unified into the same coordinate system, for example, the initial point cloud data is projected into the texture coordinate system, by using the conversion relationship between the point cloud coordinate system and the texture coordinate system. The conversion relationship can be obtained by using the calibration parameters of the lower camera system. The calibration parameters can include internal parameters such as focal length, principal point, and external parameters such as rotation matrix and translation vector.

[0072] The initial point cloud data and the initial texture data in the matching texture coordinate system are matched to obtain the initial point cloud data and the initial texture data at the same position, and a data pair is obtained, which includes corresponding point cloud data (geometric attributes such as position, normal) and texture data (color value). The above initial point cloud data at the same position is taken as a local point cloud. Color information is given to the initial point cloud data at the same position according to the data pair, and a local point cloud carrying texture information is obtained. The local point cloud carrying local information is rendered to obtain a first local color image.

[0073] In addition, there will be edge noise in the process of capturing by the monochrome camera and the color camera. After the local point cloud is determined, the point cloud at the edge can be deleted from the local point cloud to obtain a remaining data pair. Color information is enriched for the deleted local point cloud at the same position according to the remaining data pair, and a first local color image is obtained.

[0074] In step S303, based on the registration relationship between the local point cloud and the global point cloud, the first local color image is displayed on the local region corresponding to the mesh model.

[0075] In some embodiments of the present application, the scanning device can determine the registration relationship between the local point cloud and the global point cloud by using point cloud registration. The registration relationship represents the spatial transformation relationship of accurately aligning the local point cloud to the global point cloud (or the mesh model). In an embodiment, the local point cloud feature corresponding to the local point cloud and the global point cloud feature of the global point cloud are determined. The local point cloud feature and the global point cloud feature are used for coarse matching to obtain a first matching result. An algorithm such as Iterative Closest Point (ICP) is used for fine matching on the first matching result to obtain a second matching result. The registration relationship between the local point cloud and the global point cloud is determined according to the second matching result.

[0076] In the case of determining the registration relationship, the first local color image is displayed on the local region corresponding to the mesh model.

[0077] In step S304, in response to a shooting instruction, a second local color image corresponding to the local region is captured.

[0078] In some embodiments of the present application, the photographing instruction is triggered after the first partial color image is displayed on the local region corresponding to the mesh model, and the triggering manner includes at least one of automatic triggering, touch triggering, and voice triggering.

[0079] In some embodiments of the present application, after the first partial color image is displayed on the local region corresponding to the mesh model, the program of the scanning device triggers the photographing instruction to automatically collect the second partial color image, indicating that the scanning device and the scanning object are in a correct texture mapping (mapping) direction / position.

[0080] In some embodiments of the present application, the display screen (for example, the display screen 40 or the display screen 140) of the scanning device can display a plurality of controls, each of which corresponds to a different function, for example, a photographing control for collecting an image. When the local mesh corresponding to the local region is not determined (that is, when the local point cloud and the global point cloud are not registered), the photographing control on the display screen is in an inoperable state. When the local mesh corresponding to the local region is determined, the photographing control on the display screen can change from the inoperable state to the operable state. When the scanning device detects the operation (such as clicking) of the user on the photographing control, the scanning device receives a photographing instruction, so that the scanning device can collect the second partial color image corresponding to the local region by using the color camera.

[0081] In some embodiments of the present application, the scanning device further includes one or more microphones or a microphone array, which can be configured to receive the sound of the environment in which the scanning device is located, for example, a voice instruction received after the first partial color image is displayed on the local region corresponding to the mesh model. The voice instruction can be used to control the color camera of the scanning device to collect the second partial color image corresponding to the local region.

[0082] By previewing the position of the low-resolution first partial color image on the mesh model, the user can determine whether the second partial color image can be collected according to, for example, the display effect, the mapping position, and the like, so as to ensure the position and effect of the subsequent fusion of the second partial color image with the mesh model. If the scanning device detects the operation of the user clicking to collect the high-resolution image, the second partial color image is collected. The resolution of the first partial color image is lower than that of the second partial color image, which can balance the position of the subsequent fusion of the second partial color image and the resolution of the second partial color image.

[0083] It can be understood that using a low-resolution first local color image for registration and texture mapping when collecting a local point cloud can improve the frame rate of the color camera of the scanning device, so that it can be consistent with the frame rate of the monochrome camera, and better point cloud tracking registration and color texture mapping effect can be achieved. After registration and display of the color texture mapping effect, a high-resolution second local color image can be collected according to the shooting instruction, so as to ensure that the high-resolution color image meets the texture mapping effect.

[0084] Through the above embodiment, the global point cloud of the scanned object is obtained by global scanning, which can not include the texture information of the scanned object or only collect a low-resolution texture image, so as to improve the scanning efficiency and accuracy. The mesh model of the scanned object is constructed by the global point cloud, the unordered point cloud is converted into a structured surface, and a geometric framework is provided for subsequent point cloud registration. The local point cloud of the local area of the scanned object and the first local color image with low resolution are collected, the first local color image associated with the local point cloud is displayed on the local area corresponding to the mesh model through the registration relationship between the local point cloud and the global point cloud, the texture effect of the local area is provided, and the high-resolution second local color image collected under the registration can cover the local area, that is, the second local color image can be mapped to the correct position of the scanned object. Therefore, the embodiment of the present application displays the low-resolution color image on the local area by using the registration of the local point cloud and the global point cloud, and indicates the collection of the high-resolution second local color image corresponding to the local area through the above registration, so as to improve the collection efficiency and image accuracy of the second local color image.

[0085] The process of collecting the second local color image corresponding to the local area will be described below. Figure 4 As shown in Figure 4 , the display area of the scanning device includes a first display window, a second display window and a third display window, the first display window is configured to display the mesh model, the second display window is configured to display the second local color image, and the third display window is configured to display the second local black and white image corresponding to the second local color image, wherein the first display window is full-screen displayed on the display screen area, the second display window and the third display window are suspended on the first display window, and the size of the second display window and the third display window is smaller than the size of the first display window.

[0086] During the shooting process of the color camera, the scanned object can be viewed through the second display window. During the shooting process of the monochrome camera, the scanned object can be viewed through the third display window. In addition, the camera parameters of the color camera, such as brightness, white balance, etc., can be adjusted in the second display window. The shooting control, the cancel control above the shooting control and the restore control below the shooting control are also displayed on the display interface.

[0087] As Figure 4 shown, when the scanning device detects the operation of collecting the second partial color image corresponding to the local region, the mapped first partial color image is displayed on the local grid corresponding to the grid model.

[0088] In addition, a "clear" corresponding control can also be displayed on the display screen, for clearing the collected first partial color image and / or second partial color image, and a "complete" corresponding control can also be displayed on the display screen, for completing the collection of the second partial color image. When the scanning device detects that the user clicks the "complete" corresponding control, it is determined that the collection of all second partial color images of the scanned object is completed, then all second partial color images can be fused to generate a global color image of the scanned object, so as to map the global color image to the grid model constructed by the global point cloud, and complete the mapping operation of the grid model without carrying texture information.

[0089] In addition, in order to ensure the quality and efficiency of fusion, quality detection can be performed on all second partial color images before fusion. The specific detection process can be referred to the following Figure 5 embodiment.

[0090] In the process of fusion, if there is a part of the region corresponding to multiple second partial color images for any local region, it means that the multiple second partial color images are at least partially the same. Therefore, in order to ensure the resolution of mapping, the sub-image corresponding to the part of the region is segmented from the above multiple second partial color images, and the resolution of the sub-image corresponding to each part of the region is determined. The sub-image corresponding to the maximum resolution is taken as the second partial color image corresponding to any local region.

[0091] Figure 5 is a flowchart of the generation of the global color image provided by the embodiments of the present application. As Figure 5 shown, quality detection can be performed on all second partial color images to ensure the quality of each second partial color image. As Figure 5 shown, the following steps are included.

[0092] Step S501, in response to the completion of the collection of all second partial color images of the scanned object, generating a first image sequence based on the collection order of the second partial color images.

[0093] In some embodiments of the present application, as Figure 4 shown, when the scanning device detects that the user clicks the "complete" corresponding control, it is determined that the collection of all second partial color images of the scanned object is completed. The collection time stamp of each second partial color image is obtained to obtain the collection order. Based on the collection order, all second partial color images are arranged into a first image sequence.

[0094] Step S502, in response to the quality detection result of any second local color image in the first image sequence not meeting the preset requirement, determining a local region corresponding to the second local color image not meeting the preset requirement, and collecting a replacement color image.

[0095] In some embodiments of the present application, the quality detection is performed on each second local color image in the first image sequence to obtain a quality detection result corresponding to each second local color image. The quality detection includes but is not limited to detecting the resolution, definition, color accuracy, etc. of the second local color image.

[0096] In some embodiments of the present application, the preset requirement includes one or more of the preset resolution, preset definition, and color accuracy. If the quality detection result of any second local color image in the first image sequence does not meet the preset requirement, it means that the any second local color image does not meet one or more of the preset resolution, preset definition, and color accuracy. Since each local color image is derived from the collection of a corresponding local region, the scanning device can issue a prompt message for re-collection based on the corresponding local region. The prompt message can carry the second local color image not meeting the preset requirement to provide a reference for the collection position during re-collection. The scanning device uses the color image to re-collect a color image of the corresponding local region as a replacement color image.

[0097] Step S503, using the replacement color image to replace the second local color image not meeting the preset requirement to obtain a second image sequence.

[0098] In some embodiments of the present application, using the replacement color image to replace the second local color image not meeting the preset requirement, in order to save storage resources, the second local color image not meeting the preset requirement can be directly deleted after replacement to obtain the second image sequence. The present application does not limit the number of replacements of any second local color image not meeting the preset requirement, and does not limit the number of second local color images in the first image sequence.

[0099] Step S504, fusing the second image sequence to generate a global color image of the scanned object.

[0100] In some embodiments of the present application, in the case where it is determined that each second local color image meets the preset requirement, fusing the second image sequence to generate a global color image of the scanned object can guarantee the accuracy of the global color image.

[0101] Through the above embodiment, quality detection is performed on each second local color image, so that the quality of the second local color image mapped to the mesh model is ensured, and the mapping efficiency and accuracy of the mesh model are improved.

[0102] Figure 6 is a processing flowchart of the first local color image provided by the embodiment of the present application. As shown in Figure 6 , in order to clearly show the first local color image displayed in different perspectives, the first local color image is presented in the manner as shown in Figure 6 , including the following steps.

[0103] Step S601, in the first perspective, in response to an operation of collecting a second local color image corresponding to a local region in the first perspective, a first local color image in the first perspective is displayed on the local region in the first perspective.

[0104] In some embodiments of the present application, the essence of the perspective is the relative spatial pose (6 degrees of freedom) of the camera optical center and the scanned object, and the different perspectives mean the change of the camera position or direction, resulting in perspective distortion, occlusion relationship and illumination change of the imaging content. The first perspective can be any relative spatial pose of the camera optical center and the scanned object, and the current position of the color camera is set to form the first perspective of the scanned object. In the first perspective, the scanning device collects a first local color image in the first perspective by using the color camera. When the scanning device detects an operation of collecting a second local color image corresponding to a local region in the first perspective, the color camera is used to collect the second local color image corresponding to the local region in the first perspective, and a mesh model mapped with the first local color image in the first perspective is displayed on the display screen.

[0105] Step S602, in the second perspective, in response to an operation of collecting a second local color image corresponding to a local region in the second perspective, a first local color image in the second perspective is displayed on the local region in the second perspective.

[0106] In some embodiments of the present application, the operation process of step S601 is the same, and the related process in the second perspective will not be described again. The first local color image displayed in the first perspective and the second local color image displayed in the second perspective are at least partially different, and the time of mapping the first local color image in the first perspective is earlier than the time of mapping the first local color image in the second perspective.

[0107] Step S603, in the case that the local region in the first perspective and the local region in the second perspective do not completely overlap, a mesh model mapped with the first local color image in the second perspective is displayed through the first display window.

[0108] In some embodiments of the present application, in order to display the first partial color image of the completed mapping in real time, if the local region under the first view and the local region under the second view do not completely overlap, the grid model mapped with the second partial color image under the second view can be displayed through the first display window, which is full-screen displayed on the display screen.

[0109] In step S604, the first partial color image under the first view is displayed through the floating window in the first display window.

[0110] In some embodiments of the present application, the floating window is suspended on the first display window, and the size of the floating window is smaller than the size of the first display window. The first partial color image under the first view is displayed through the floating window, so that the first partial color image collected under the first view can also be presented under the second view.

[0111] The display of the first display window and the floating window will be described below. Figure 7 As shown in FIG. 6A, the grid model mapped with the first partial color image under the second view is displayed through the first display window. Figure 7 As shown in FIG. 6B, there are multiple first partial color images under the first view, and multiple floating windows are used to display each first partial color image under the first view. Figure 7

[0112] Through the above embodiments, the mapping of the first partial color image is intuitively displayed in the form of visualization and floating chart. Since the collection of the first partial color image and the collection of the second partial color image have relevance, all the collected first partial color images can be presented through visualization, which can improve the efficiency of collecting the second partial color image to a certain extent.

[0113] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program includes program instructions, and the method implemented by the program instructions will be described with reference to the above-mentioned embodiments of the present application.

[0114] The computer readable storage medium can be the internal memory of the electronic device, such as the hard disk or the memory of the electronic device. The computer readable storage medium can also be the external storage device of the electronic device, such as the plug-in hard disk, the smart media card (SMC), the secure digital (SD) card, the flash card, etc.

[0115] ​In some embodiments, the computer readable storage medium can include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required by at least one function, and the like; and the data storage area can store data created according to the use of the electronic device, and the like.

[0116] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0117] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented by other ways. For example, the apparatus / terminal device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0119] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected to achieve the purpose of the embodiments according to actual needs.

[0120] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image processing method applied to a scanning device, wherein, The method comprises: performing global scanning on a scanned object to obtain a global point cloud, and constructing a mesh model of the scanned object based on the global point cloud; collecting a local point cloud and a first local color image of a local region of the scanned object; based on a registration relationship between the local point cloud and the global point cloud, the first local color image is displayed on the local region corresponding to the mesh model, and whether to collect a second local color image corresponding to the local region is determined according to a display effect and a mapping position of the first local color image on the mesh model, if yes, generating a shooting instruction; and in response to the shooting instruction, collecting the second local color image corresponding to the local region, wherein a resolution of the first local color image is lower than a resolution of the second local color image; in response to completion of collection of all second local color images of the scanned object, fusing all the second local color images to generate a global color image of the scanned object; mapping the global color image to the mesh model constructed based on the global point cloud.

2. The method of claim 1, wherein, Further comprising: completing collection of the local point cloud, the first local color image and the corresponding second local color image of the scanned object under different viewing angles.

3. The method of claim 1, wherein, The operation of fusing all the second local color images to generate the global color image of the scanned object in response to completion of collection of all second local color images of the scanned object comprises: in response to completion of collection of all second local color images of the scanned object, generating a first image sequence based on a collection sequence of the second local color images; in response to a quality detection result of any second local color image in the first image sequence not meeting a preset requirement, determining a local region corresponding to the second local color image not meeting the preset requirement, and collecting a replacement color image; using the replacement color image to replace the second local color image not meeting the preset requirement to obtain a second image sequence; fusing the second image sequence to generate the global color image of the scanned object.

4. The method of claim 3, wherein, The second local color image collected later will cover a common region between the second local color image collected earlier and the second local color image collected later.

5. The method of claim 1, wherein, The display region of the scanning device comprises a first display window, a second display window and a third display window, the first display window is configured to display the mesh model, the second display window is configured to display the second local color image, and the third display window is configured to display a second local black-and-white image corresponding to the second local color image, wherein the first display window is displayed full screen on the display region, the second display window and the third display window are suspended on the first display window, and sizes of the second display window and the third display window are smaller than a size of the first display window.

6. The method of claim 1, wherein, The first local color image is mapped to the local point cloud.

7. The method of claim 1, wherein, The shooting instruction is triggered after the first partial color image is displayed on the local region corresponding to the mesh model, and the triggering manner includes at least one of automatic triggering, touch triggering, and voice triggering.

8. A scanning device, wherein, The scanning device includes a processor and a memory, the memory stores a computer program, and the processor implements the image processing method according to any one of claims 1-7 when executing the computer program.

9. A scanning device, wherein, Comprise: a projector configured to project light rays to a scanned object; one or more monochrome cameras configured to collect reflected light of the scanned object for the light rays; one or more color cameras configured to collect color images of the scanned object; a processor configured to: obtain a global point cloud of the scanned object based on the reflected light, and construct a mesh model of the scanned object through the global point cloud; collect a local point cloud of a local region of the scanned object based on the reflected light, and collect a first partial color image of the local region using the one or more color cameras; display the first partial color image on the local region corresponding to the mesh model based on a registration relationship between the local point cloud and the global point cloud, and determine whether to collect a second partial color image corresponding to the local region according to a display effect and a mapping position of the first partial color image on the mesh model, and generate a shooting instruction if so; in response to the shooting instruction, collect a second partial color image corresponding to the local region using the one or more color cameras, wherein the resolution of the first partial color image is lower than that of the second partial color image; in response to completion of collection of all second partial color images of the scanned object, fuse all the second partial color images to generate a global color image of the scanned object; map the global color image to the mesh model constructed by the global point cloud.

10. A computer readable storage medium, wherein, The computer readable storage medium stores at least one instruction, and the at least one instruction is executed by the processor to implement the image processing method according to any one of claims 1-7.

Images