Image processing method and apparatus, and device and storage device

Abstract

Provided in the embodiments of the present disclosure are an image processing method and apparatus, and a device and a storage medium. During the process of scanning a target object by using a three-dimensional scanning device, a three-dimensional model obtained by means of performing real-time three-dimensional reconstruction on the target object on the basis of data collected by the three-dimensional scanning device can be acquired; moreover, a real-scene image of the target object can be collected by means of a camera on an AR device, the three-dimensional model is projected into the real-scene image on the basis of a matching relationship between the three-dimensional model and the target object in the real-scene image, so as to determine a scanned region in the real-scene image, and the scanned region is identified, and thus a first augmented-reality image is obtained; and then, the first augmented-reality image can be displayed by means of a display screen of the AR device, such that by means of viewing the first augmented-reality image, a user can not only see real-time situations of the target object and the three-dimensional scanning device, but can also see the current scanning situation of the target object, and there is no requirement to switch a line of sight back and forth during scanning, and operation is very convenient.

Description

Image processing method, device, equipment and storage device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 202311198534.5 and application name “Three-dimensional scanning method, device, equipment and storage medium”, and the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 202311202126.2 and application name “Image processing method, device, equipment and storage medium”, all of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0003] With the development of 3D scanning technology, it has been gradually applied to various fields such as medicine, industry, and archaeology, and 3D scanning equipment suitable for various fields has gradually emerged. Currently, when using 3D scanning equipment to scan a target object and reconstruct a 3D model of the target object, the 3D scanning equipment usually needs to be connected to a terminal device with a display screen, such as a laptop or tablet computer, and the collected data is sent to these terminal devices in real time. These terminal devices then perform real-time 3D reconstruction of the target object based on the collected data, obtaining a 3D model of the target object and displaying the currently reconstructed 3D model on the display screen.

[0004] When a user uses a 3D scanning device to scan a target object, the user needs to adjust the position, movement speed, etc. of the 3D scanning device in real time based on the reconstruction of the current 3D model. Therefore, the user's line of sight needs to constantly switch back and forth between the 3D scanning device and the display screen of the terminal device, which brings a lot of inconvenience to the user and affects the user experience.

[0005] In addition, AR (Augmented Reality) technology can enhance the real world by superimposing virtual information onto the real world. In some application scenarios, some virtual images need to be superimposed on each frame of real-scene images in the real-time video stream captured by the AR device to obtain an augmented reality image video stream so that users can obtain an augmented reality image video stream with richer and more comprehensive information. Currently, when performing the superposition process, the superposition of each frame of real-scene image and virtual image is processed independently. For example, for the current frame of real-scene image, the matching relationship between the virtual image and the target object in the real-scene image can be determined based on the internal and external parameters of the AR device's camera, and the two are aligned based on the matching relationship, and the virtual image is superimposed on the real-scene image. For the next frame of real-scene image, the superposition of the two is completed in a similar manner. Since there is a certain matching error when determining the matching relationship between the virtual image and each frame of real-scene image, the video stream of the augmented reality image finally displayed will have jumps and jitters.

[0006] Summary of the Invention

[0007] The present disclosure provides an image processing method, apparatus, device, and storage medium.

[0008] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0009] In the process of scanning the target object using the three-dimensional scanning device, a three-dimensional model of the target object is obtained in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0010] Acquire the real-scene image of the target object captured by the AR device in real time;

[0011] Determining a matching relationship between the three-dimensional model and the target object in the real scene image, and projecting the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image;

[0012] Marking the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through an AR device;

[0013] A preset image is displayed in the AR device, where the preset image includes: a first augmented reality image.

[0014] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, wherein a real scene image includes a current frame real scene image in a video stream collected from an AR device, and the method further includes:

[0015] Acquire a virtual image and a current frame real scene image; the virtual image includes a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by a three-dimensional scanning device, or pre-constructed image data including the target object;

[0016] Determining an initial matching relationship between the virtual image and the current frame of the real scene image, and optimizing the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images preceding the current frame of the real scene image in the video stream;

[0017] Superimposing the virtual image onto the current frame real scene image based on the target matching relationship to obtain a second augmented reality image, so as to display the second augmented reality image on the AR device;

[0018] A preset image is displayed in the AR device, where the preset image includes: a first augmented reality image, and / or a second augmented reality image, and / or an image obtained by fusing the first augmented reality image and the second augmented reality image.

[0019] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0020] a three-dimensional model acquisition module configured to acquire a three-dimensional model of the target object in real time during a process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0021] A real scene image acquisition module is configured to acquire a real scene image of a target object captured by an AR device in real time;

[0022] a projection module configured to determine a matching relationship between the three-dimensional model and a target object in the real image, and project the three-dimensional model into the real image based on the matching relationship to determine a scanned area in the real image;

[0023] an identification module that identifies the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through an AR device;

[0024] The display module is configured to display a preset image in the AR device, where the preset image includes: a first augmented reality image.

[0025] According to a second aspect of an embodiment of the present disclosure, an image processing apparatus is provided, wherein a real scene image includes a current frame real scene image in a video stream collected from an AR device, and the apparatus further includes:

[0026] an acquisition module configured to acquire a virtual image and a current frame real scene image; the virtual image includes a three-dimensional model obtained by real-time three-dimensional reconstruction of a target object based on data collected by a three-dimensional scanning device, or pre-constructed image data including the target object;

[0027] a matching module configured to determine an initial matching relationship between the virtual image and the current frame of the real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images preceding the current frame of the real scene image in the video stream;

[0028] an overlay module configured to overlay the virtual image onto the current frame real scene image based on the target matching relationship to obtain a second augmented reality image, so as to display the second augmented reality image on the AR device;

[0029] The display module is also configured to display a preset image in the AR device, and the preset image includes: a first augmented reality image, and / or a second augmented reality image, and / or an image obtained by fusion of the first augmented reality image and the second augmented reality image.

[0030] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes a processor, a memory, and computer instructions stored in the memory for execution by the processor. When the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0031] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0032] In the disclosed embodiment, while a user is scanning a target object using a 3D scanning device, a 3D model of the target object can be obtained by real-time 3D reconstruction of the target object based on data collected by the 3D scanning device. Simultaneously, a real-life image of the target object can be captured by a camera on an AR device, and a matching relationship between the 3D model and the target object in the real-life image can be determined. Based on the matching relationship, the 3D model is projected onto the real-life image to determine a scanned area in the real-life image, and the scanned area in the real-life image is identified to obtain a first augmented reality image. The first augmented reality image can then be displayed on the display screen of the AR device. By viewing the first augmented reality image, the user can see both the real-time status of the target object and the 3D scanning device, as well as the current scanning status of the target object. Therefore, the user can determine how to adjust the position of the 3D scanning device and determine the scanning method simply by viewing the content displayed on the AR device display screen, without having to switch their line of sight back and forth during the scanning process. This makes the operation very convenient, enhancing the user's experience during the scanning process.

[0033] In addition, in the embodiment of the present disclosure, in the process of superimposing the virtual image on the real-time video stream of the target object, when determining the matching relationship between the virtual image and the current frame real-scene image in the video stream, the matching relationship between the virtual image and the one or more frames of real-scene images before the current frame real-scene image can be jointly determined, that is, the matching relationship between the virtual image and the current frame real-scene image in the video stream is constrained and optimized through the matching relationship between the virtual image and the previous one or more frames of real-scene images, which is equivalent to smoothing the matching relationship between the virtual image and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the finally generated second augmented reality image, and improving the display effect of the video stream of the second augmented reality image.

[0034] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0036] FIG1 is a schematic diagram of a real-time scanning of a target object using a three-dimensional scanning device according to an embodiment of the present disclosure.

[0037] FIG2 is a schematic diagram of an application scenario of an embodiment of the present disclosure.

[0038] FIG3 is a flow chart of an image processing method according to an embodiment of the present disclosure.

[0039] FIG4 is a schematic diagram of identifying a scanning trajectory line in a first augmented reality image according to an embodiment of the present disclosure.

[0040] FIG5 is a schematic diagram of identifying scanning quality in a first augmented reality image according to an embodiment of the present disclosure.

[0041] FIG6 is a schematic diagram of identifying a specific type of area in a first augmented reality image according to an embodiment of the present disclosure.

[0042] FIG7 is a schematic diagram of an embodiment of the present disclosure showing a tooth currently under discussion in a first augmented reality image displayed by AR glasses worn by a doctor and a patient.

[0043] FIG8 is a schematic diagram of an embodiment of the present disclosure in which a three-dimensional tooth model is superimposed on a real-scene image video stream with a jump problem.

[0044] FIG9 is a schematic diagram of determining a matching relationship according to an embodiment of the present disclosure.

[0045] FIG10 is a flow chart of another image processing method according to an embodiment of the present disclosure.

[0046] FIG11 is a schematic diagram of the logical structure of an image processing device according to an embodiment of the present disclosure.

[0047] FIG12 is a schematic diagram of the logical structure of another image processing device according to an embodiment of the present disclosure.

[0048] FIG13 is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0050] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a," "an," and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality.

[0051] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0052] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0053] Currently, when using a 3D scanning device to scan a target object and reconstruct a 3D model of the target object, the 3D scanning device usually needs to be connected to a terminal device with a display screen, such as a laptop or tablet computer, and the collected data is sent to these terminal devices in real time. These terminal devices then perform real-time 3D reconstruction of the target object based on the collected data to obtain a 3D model of the target object and display the currently reconstructed 3D model on the display screen. Taking an oral scanning scenario as an example, as shown in Figure 1, the oral scanning device can usually be connected to the doctor's computer. When the doctor uses the oral scanning device to scan the patient's oral cavity, the oral scanning device can send the collected data to the computer in real time. The computer then performs real-time 3D reconstruction of the patient's oral cavity based on the collected data to obtain a reconstructed 3D model of the oral cavity and displays the real-time reconstructed 3D model on the computer screen.

[0054] When scanning a target object with a 3D scanning device, the user needs to determine the current scanning status of the target object based on the reconstruction status of the current 3D model, namely, which areas have been scanned, which areas have not been scanned, and the scan quality of the scanned areas, so as to adjust the position and scanning speed of the 3D scanning device in real time. Therefore, the user's line of sight needs to constantly switch back and forth between the target object and the display screen of the terminal device, causing many inconveniences to the user and affecting the user experience. For example, as shown in Figure 1, the doctor needs to check the real-time reconstructed 3D model displayed on the display screen, adjust the scanning method based on the reconstruction status of the 3D model, and then look at the patient's mouth to adjust the position and scanning speed of the oral scanning device. The constant switching of the line of sight between the patient's mouth and the display screen is very cumbersome.

[0055] In view of this, the present disclosure provides an image processing method that can combine AR (Augmented Reality) technology with 3D scanning technology. When a user uses a 3D scanning device to scan a target object, a 3D model of the target object can be obtained by real-time 3D reconstruction of the target object based on data collected by the 3D scanning device. At the same time, a real-life image of the target object can be captured in real time by a camera on the AR device, and a matching relationship between the 3D model and the target object in the real-life image can be determined. Based on the matching relationship, the 3D model is projected onto the real-life image to determine the scanned area in the real-life image. The scanned area is then marked to obtain a first augmented reality image, which can then be displayed on the display screen of the AR device. Thus, by viewing the first augmented reality image, the user can see both the real-time status of the target object and the 3D scanning device, as well as the current scanning status of the target object. Therefore, the user can determine how to adjust the position of the 3D scanning device and determine the scanning method simply by viewing the content displayed on the AR device display, without having to switch their line of sight back and forth during the scanning process. This operation is very convenient and improves the user experience during the scanning process.

[0056] For example, as shown in Figure 2, which is a schematic diagram of an application scenario of an embodiment of the present disclosure, a user can hold a handheld oral scanning device to perform a three-dimensional scan of the patient's oral cavity. At the same time, the user can wear AR glasses. The AR glasses can obtain the data collected by the oral scanning device, and use the obtained data to reconstruct a three-dimensional model in real time, and project the three-dimensional model into the real-life image of the oral cavity collected by the AR glasses, determine which area of ​​the real-life image of the oral cavity has been scanned, and then mark the scanned area to obtain a second augmented reality image, and display it on the display screen of the AR glasses. By viewing the content displayed by the AR glasses, the user can understand the current oral scanning status and adjust the scanning method based on the scanning status.

[0057] AR technology can enhance the real world by overlaying virtual information onto it. In some application scenarios, it is necessary to superimpose virtual images onto the real-world images in the real-time video stream captured by the AR device to generate an augmented reality image video stream, so that users can obtain a richer and more comprehensive augmented reality image video stream.

[0058] For example, for some scenarios, users need to add some virtual information to the real scene to make it more realistic. For example, when a user takes a selfie, virtual glasses or a virtual hat are automatically added to the user's face image.

[0059] For example, in a scenario where a user performs a real-time three-dimensional scan of a target object, in order to facilitate the user to understand the current scanning status, the data collected by the three-dimensional scanning device can be used for real-time three-dimensional reconstruction to obtain a three-dimensional model of the target object. At the same time, the AR device can be used to collect a video stream of the target object in real time, and then the real-time reconstructed three-dimensional model can be superimposed on each frame of the real-scene image of the video stream to obtain an augmented reality image video stream. In this way, the user can know the real-time scanning status of the target object by viewing the augmented reality image video stream, thereby guiding the user to perform a three-dimensional scan of the target object.

[0060] For example, in some scenarios, users can only see the surface of the target object and cannot see the internal situation of the target object. In order to allow users to see the internal situation of the target object when viewing the target object, a three-dimensional model of the target object can be reconstructed in advance. Then, the AR device can be used to capture the video stream of the target object in real time, and the pre-reconstructed three-dimensional model can be superimposed on each frame of the real-life image in the video stream to obtain an augmented reality image video stream. That is, the user can know the internal situation of the target object by viewing the augmented reality image video stream. Taking the oral cavity as an example, doctors can usually only see the teeth and gums in the oral cavity and cannot see the tooth roots. If a three-dimensional model of the tooth roots is pre-constructed, when the doctor examines the patient's oral cavity, the doctor can wear AR glasses. The AR glasses can capture the real-life image of the patient's oral cavity in real time, and then superimpose the three-dimensional model of the tooth roots on each frame of the captured real-life image to obtain an augmented reality image, which is then displayed on the AR glasses' display screen, so that the doctor can clearly understand the condition of the root area of ​​each tooth.

[0061] For the above-mentioned scenarios or similar scenarios, since the virtual image needs to be superimposed on each frame of the real-scene image in the real-time video stream collected by the AR device, the superposition of each frame of the real-scene image and the virtual image is currently processed independently during the superposition process. For example, for the current frame of the real-scene image, the matching relationship between the virtual image and the real-scene image can be determined based on the internal and external parameters of the AR device's camera, and the two can be aligned based on the matching relationship, and the virtual image can be superimposed on the real-scene image. For the next frame of the real-scene image, a similar method is used to complete the superposition of the two. Since there is a certain matching error when determining the matching relationship between the virtual image and each frame of the real-scene image, the final displayed augmented reality image video stream will have jump and jitter problems.

[0062] For example, as shown in Figure 8, taking the target object as the oral cavity, when performing real-time three-dimensional scanning of the oral cavity, in order to facilitate the user to understand the current scanning situation, the oral cavity three-dimensional model reconstructed in real time based on the data collected by the three-dimensional scanning device can be superimposed on the real-scene image of the oral cavity. Superposition refers to projecting the three-dimensional model into the real-scene image, determining the scanned area, and marking the scanned area, so that the user can understand which areas are currently scanned areas.

[0063] When superimposing the 3D model of the oral cavity onto the real-life image video stream, for example, for the current frame of the real-life image, errors in the determined matching relationship may cause the teeth in the 3D model to not be fully aligned with those in the current frame of the real-life image. Consequently, when the 3D model is projected onto the current frame of the real-life image, the teeth in the projected area and the real-life image are also misaligned. For example, the teeth in the 3D model are offset to the left by a certain distance compared to those in the real-life image. Furthermore, for the next frame of the real-life image, errors in the determined matching relationship may cause the teeth in the 3D model to be offset to the right by a certain distance compared to those in the real-life image. Consequently, when the user sees the two frames of the final augmented reality image, the position of the 3D model may suddenly jump from the left to the right of the image, resulting in a jump. Overall, the video stream of the augmented reality image viewed by the user is constantly jittering, severely impacting the display quality of the augmented reality video stream.

[0064] The scenario of superimposing other virtual images onto the real-life image video stream is similar. For example, superimposing virtual glasses on a real-time facial image may cause the virtual glasses to be always positioned to the left or right of the face due to matching errors, causing the video stream of the augmented reality image seen by the user to be constantly jittery, seriously affecting the display effect of the augmented reality image video stream.

[0065] Based on this, an embodiment of the present disclosure provides an image processing method, as shown in Figure 9. In the process of superimposing a virtual image on a real-time video stream captured by an AR device, when determining the matching relationship between the virtual image and the current frame real-scene image in the video stream, the matching relationship between the virtual image and the one or more frames of real-scene images before the current frame real-scene image can be jointly determined. That is, the matching relationship between the virtual image and the previous one or more frames of real-scene images is constrained and optimized through the matching relationship between the virtual image and the current frame real-scene image in the video stream, which is equivalent to smoothing the matching relationship between the virtual image and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the finally generated second augmented reality image, and improving the display effect of the video stream of the second augmented reality image.

[0066] In some scenarios, the method of the embodiments of the present disclosure can be executed by an AR device. For example, the AR device can be connected to a three-dimensional scanning device for communication, and acquire data collected by the three-dimensional scanning device in real time from the three-dimensional scanning device, and then use the acquired data to reconstruct a three-dimensional model of the target object. At the same time, the AR device can use a camera to collect a real-life image of the target object in real time, and project the three-dimensional model onto the currently collected real-life image to obtain and display a first augmented reality image. For another example, the AR device can acquire a virtual image, and at the same time, the AR device can use a camera to collect a video stream in real time, determine the matching relationship between the virtual image and each frame of the real-life image in the video stream, and then align and superimpose the two based on the matching relationship to obtain and display a second augmented reality image.

[0067] In some scenarios, the method of the embodiment of the present disclosure may also be executed by other devices connected to the AR device, which may be devices with good processing performance such as laptops, tablet computers, and cloud servers. These devices can obtain data collected by the three-dimensional scanning device, and use these data to perform three-dimensional reconstruction of the target object to obtain a three-dimensional model. At the same time, they can obtain the real-scene image collected by the AR device, project the three-dimensional model into the real-scene image, obtain a first augmented reality image, and then send it to the AR device, which displays the first augmented reality image. These devices can also obtain a virtual image. At the same time, they can obtain the video stream collected by the AR device in real time, determine the matching relationship between the virtual image and the real-scene image of each frame in the video stream, and then align and superimpose the two based on the matching relationship to obtain a second augmented reality image, and then send it to the AR device, which displays the second augmented reality image.

[0068] In some scenarios, the method of the embodiments of the present disclosure may also have some steps performed by the AR device and some steps performed by other terminal devices. For example, the step of 3D reconstruction to obtain a 3D model is performed by other terminal devices, while the step of projecting the 3D model onto the real scene image to generate and display the first augmented reality image is performed by the AR device. For another example, the step of determining the matching relationship is performed by other devices, while the step of superimposing and displaying the second augmented reality image is performed by the AR device.

[0069] Among them, the three-dimensional scanning equipment can be handheld or fixed scanners such as oral scanners, facial scanners, ear scanners, industrial scanners, and professional scanners, which can realize three-dimensional reconstruction of objects or scenes such as teeth, faces, ears, human bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical devices, and buildings.

[0070] The specific setting of the execution entity can be flexibly set based on actual needs, and the embodiments of the present disclosure do not limit it.

[0071] The target object in the embodiments of the present disclosure can be any object that requires three-dimensional reconstruction, such as an oral cavity, a building, a cultural relic, etc. Correspondingly, the three-dimensional scanning device in the embodiments of the present disclosure can also be any type of three-dimensional scanning device, such as an oral cavity scanning device, an industrial scanning device, etc. The virtual image in the embodiments of the present disclosure can be any information that requires superimposing a real scene, such as a two-dimensional image, a three-dimensional model, etc., such as an oral cavity, a human organ, etc., and the embodiments of the present application are not limited thereto.

[0072] The AR device in the embodiments of the present disclosure can be various types of AR devices, such as AR glasses, AR displays, etc. The AR device includes a camera and a display screen, wherein the camera is configured to capture a real-life image of a target object, and the display screen is configured to display the ultimately generated first augmented reality image and second augmented reality image. In addition, the AR device may also include various sensors configured to determine the current position of the user's line of sight.

[0073] As shown in FIG3 , the method may include the following steps:

[0074] S302. In the process of scanning the target object using the three-dimensional scanning device, a three-dimensional model of the target object is obtained in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device.

[0075] In step S302, as the user scans the target object using a 3D scanning device, a 3D model of the target object is acquired in real time. This 3D model can be reconstructed in real time based on the data collected by the 3D scanning device. For example, 3D reconstruction can be performed on all data collected at the current moment at regular intervals (e.g., 0.5 seconds) to obtain the current 3D model. As more data is collected over time, the real-time reconstructed 3D model becomes increasingly complete.

[0076] S304: Acquire a real-scene image of the target object captured by the AR device in real time.

[0077] In step S304, a real-life image of the target object captured by the AR device can be acquired in real time. The AR device may include a camera, which can be used to capture a real-life image including the target object. Furthermore, the relative positional relationship between the three-dimensional scanning device and the target object can be determined based on the real-life image. Of course, since the real-life image also changes in real time, a frame of the real-life image can also be captured at fixed intervals.

[0078] S306 : Determine a matching relationship between the three-dimensional model and the target object in the real scene image, and project the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image.

[0079] In step S306, after acquiring the current 3D model and real-world image, the matching relationship between the 3D model and the target object in the real-world image can be determined. This matching relationship refers to the correspondence between 3D points in the 3D model and pixels in the real-world image. Because the target objects in both the 3D model and the real-world image correspond to real-world target objects, their coordinate systems can be unified and converted to the same coordinate system. This conversion relationship is the matching relationship between the two. Based on this matching relationship, the currently reconstructed 3D model can then be projected into the real-world image to determine the projection area of ​​the 3D model in the real-world image. This projection area is the scanned area in the current real-world image.

[0080] S308: Mark the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through an AR device.

[0081] Typically, when a user is performing a three-dimensional scan of a target object, the user is more concerned about which areas of the current target object have been scanned and which areas have not been scanned, so that the user can perform a supplementary scan on the unscanned areas and collect data on the unscanned areas. In order to facilitate the user to intuitively know which areas are scanned areas, in step S308, the scanned areas in the real scene image can be marked to obtain a first augmented reality image. There are many ways to mark the areas, as long as the user can distinguish which areas have been scanned. For example, the scanned areas and the unscanned areas can be rendered in different colors, or the scanned areas can be framed out, or the projection points of the three-dimensional model when projected onto the area can be displayed in the scanned area, wherein the density of the projection points can indicate the density of the three-dimensional model corresponding to the area. After obtaining the first augmented reality image after the marking process, the first augmented reality image can be displayed in real time through the AR device, so that the user can understand the current real-time scanning status of the target object by viewing the first augmented reality image. Considering that there are often corresponding optimal scanning trajectories for different types of target objects during the scanning process, scanning the target object based on the optimal scanning trajectory can not only perform a comprehensive scan of the target object, but also improve the scanning efficiency.

[0082] In some embodiments, when marking the scanned area in the real-life image to obtain the first augmented reality image, the three-dimensional model can be projected into the real-life image to obtain a projected image, which is then rendered into different colors and superimposed on the real-life image to identify the scanned area. The first augmented reality image can include multiple layers, and the scanning trajectory, the real-life image, and the projected image of the three-dimensional model can be located on different layers. When the three-dimensional model is superimposed on the real-life image, to prevent the three-dimensional model from obstructing the scanning trajectory, the layer containing the scanning trajectory can be placed on the top layer.

[0083] Furthermore, when scanning a target object, users are often concerned about scan quality. For example, they may want to determine whether any areas of the target object are missed, whether the scan density of each area meets the required standards, or whether the reconstructed 3D points in some areas are sparse and lack precision. To allow users to understand the scan quality of each currently scanned area in real time, in some embodiments, the scanned area can be divided into multiple sub-areas. These sub-areas can be divided equally or unevenly. The scan quality of each sub-area can then be determined and indicated in the real-life image, allowing users to intuitively understand the scan status of the currently scanned area. For example, after reconstructing a 3D model based on data collected by a 3D scanning device, the matching relationship between the 3D model and the target object in the real-life image can be determined. Based on the matching relationship, the number and density of 3D points corresponding to each sub-area of ​​the target object in the real-life image can then be determined. A denser 3D point density indicates a higher scan density and better scan quality for that area. In some scenarios, the scan quality of each sub-area can be indicated by color depth. For example, a darker color indicates a denser 3D point density and better scan quality for that sub-area. As shown in Figure 5, the color depth can be used to indicate the scanning quality. Darker teeth indicate that the reconstructed 3D points are denser, and lighter teeth indicate that the reconstructed 3D points are sparser and need to be re-scanned. In some scenarios, some points can also be covered in each sub-area. The denser the covered points, the denser the 3D points reconstructed after scanning the sub-area, and the better the scanning quality. Of course, the specific identification method can be flexibly selected based on actual needs, as long as the user can intuitively understand the scanning quality of each sub-area of ​​the scanned area based on the first augmented reality image finally presented, so that the user can re-scan those sub-areas with poor scanning quality, thereby collecting more and more complete data.

[0084] Generally speaking, during the scanning process, users may pay more attention to some specific types of areas of the target object, so these areas can be marked out for the convenience of users to view. For example, in some scenarios, the target object is the oral cavity, and the specific type of area may be the diseased area in the oral cavity, such as caries. Doctors usually pay more attention to these parts of the oral cavity, so these areas can be marked out. In some scenarios, the target object may be a pipe, and the specific type of area may be an area with cracks in the pipe, or an area with rust and corrosion in the pipe. Of course, for different scenarios, specific areas can be flexibly set based on actual needs.

[0085] In order to allow users to intuitively view specific types of areas during the scanning process, in some embodiments, after obtaining the real-scene image captured by the AR device, the real-scene image can also be recognized. For example, a pre-trained image recognition model can be used to recognize the real-scene image to determine whether the specific type of area exists in the real-scene image. If so, the specific type of area is identified. For example, as shown in Figure 6, the area can be framed or rendered into a specific color, etc. The specific identification method can be flexibly set based on actual needs.

[0086] When projecting a three-dimensional model onto a real-life image to determine the current scanned area, accurately determining the matching relationship between the three-dimensional model and the target object in the real-life image is a prerequisite for obtaining an augmented reality image with better results. In some embodiments, if the target object is an oral cavity, in order to obtain a more accurate matching relationship, the matching relationship can be constrained or corrected with the help of the correspondence between the teeth in the three-dimensional model and the teeth in the real-life image to obtain a more accurate matching relationship. For example, the three-dimensional model can be semantically segmented to obtain multiple tooth models, and the oral cavity area in the real-life image can be semantically segmented to obtain multiple tooth areas. Among them, when the three-dimensional model is semantically segmented, the gum area and each tooth in the three-dimensional model of the oral cavity can be identified to obtain a gum model and multiple tooth models. Similarly, when the real-life image is semantically segmented, the gum area and each tooth in the image can also be identified to obtain a gum area and multiple tooth areas.

[0087] Because these multiple tooth models correspond to individual teeth in a real mouth, and the tooth regions also correspond to individual teeth in a real mouth, the correspondence between the multiple tooth models and the multiple tooth regions can be determined based on the tooth numbers. This one-to-one correspondence can then be used as a constraint to determine the matching relationship between the 3D model and the target object in the real-life image. By adding the correspondence between the teeth as a constraint, the resulting matching relationship is more accurate.

[0088] For example, after obtaining a tooth model and tooth region corresponding to a real-world tooth, we can determine easily extractable feature lines (e.g., the upper edge line) or easily identifiable feature points (e.g., the center point or the highest point of a protrusion) from the tooth model and tooth region. Correspondence between these feature lines or points in the tooth model and tooth region is established as a constraint to determine the matching relationship between the 3D model and the target object in the real-life image.

[0089] In some embodiments, to accurately determine matching feature lines or feature points from the tooth model and tooth region, the feature line can be the boundary between the teeth and gums. Since individual teeth and gum regions can be identified during semantic segmentation, and the boundary between the teeth and gums is relatively clear and easy to extract, the tooth model and tooth region corresponding to the same tooth can be determined from the 3D model and the current frame real-life image, respectively. The boundary between the teeth and gums can then be extracted from both, and the matching relationship can be determined using the corresponding relationship between the boundary lines as a constraint.

[0090] For example, when using the correspondence between teeth as a constraint to determine a matching relationship, a tooth model and tooth region corresponding to the same tooth can be determined based on the correspondence between multiple tooth models and tooth regions. Then, the boundary between the tooth and gum can be extracted from the tooth model, hereinafter referred to as the first boundary line. At the same time, the boundary between the tooth and gum can be extracted from the tooth region, hereinafter referred to as the second boundary line. The first boundary line and the second boundary line correspond to the boundary between the tooth and gum in the same real tooth. The first boundary line can then be projected into the real-life image to obtain a projection line, and a reprojection error between the projection line and the second boundary line can be constructed. The matching relationship can then be solved using the minimization of the reprojection error as a constraint.

[0091] Since the boundary between teeth and gums is relatively clear and easy to extract, the tooth model and tooth area corresponding to the same tooth can be determined from the three-dimensional model and the real-life image respectively, and then the boundary between teeth and gums can be extracted from both respectively. The above boundary line is used to determine the reprojection error, and then the reprojection error is used as a constraint to solve the matching relationship, so that the final matching relationship is more accurate.

[0092] For example, suppose the 3D scanning device is at t i The three-dimensional model of the teeth scanned at each moment is in the coordinate system of the three-dimensional scanning device. AR devices in t i The real scene image collected at each moment is The tooth position number can be identified and segmented on the real image to obtain the tooth area set corresponding to each tooth Where k represents the tooth position number.

[0093] Similarly, the tooth 3D model can also be segmented and identified to obtain a 3D mesh subset belonging to each tooth. in represents the mesh subset belonging to tooth position k in the tooth model (the three-dimensional model of each tooth). The boundary line of the tooth is the boundary between the tooth and the gum, recorded as It is also a part with special semantics.

[0094] Through the AR device camera internal and external parameter matrix K, T i , the 3D point P in the 3D tooth model can be projected onto the real-scene image plane captured by the camera to obtain the image coordinates of the projection point: u = KT i P. The internal parameter matrix K is only related to the optical system of AR glasses and is constant during the shooting process. The external parameter matrix T i It is the item to be solved, which is configured to describe the relative position relationship between the 3D scanning device and the camera, and accurately solve the external parameter matrix T i It's the key, KT i This is the aforementioned matching relationship.

[0095] Solving the extrinsic parameter matrix requires finding the correspondence between the teeth in the 3D model and the teeth in the 2D image, which can be done with the help of the tooth boundary lines. After finding the correspondence between the tooth boundary lines in the 3D model and the 2D image, and determining the reprojection error between the two, the extrinsic parameter matrix can be solved with the minimization of the reprojection error as a constraint, as shown in formula (1):

[0096] in, is the pixel coordinate of the jth tooth and gum boundary line in the image at time ti, is the pixel coordinate of the jth tooth-gingival boundary line in the image at time ti-1. is the three-dimensional coordinate of the jth tooth-gingival boundary line in the three-dimensional model at time ti (i.e., the tooth-tooth boundary line of a certain tooth model), is the 3D coordinate of the jth tooth and gum boundary line in the 3D model at time ti-1, K is the camera intrinsic parameter matrix, T i is the external parameter matrix of the camera at time ti, T i-1 is the external parameter matrix of the camera at time ti-1, and ω1 and ω2 are preset coefficients.

[0097] When determining the extrinsic parameter matrix, by introducing the reprojection error of the tooth-gingival boundary line as a constraint condition, the final matching relationship can be made more accurate.

[0098] In some embodiments, the three-dimensional scanning device may include two scanning modes, namely a high-precision scanning mode and a low-precision scanning mode. In the high-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is small, that is, the measurement error is small. In the low-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is large, that is, the measurement error is large. Considering that certain areas of the target object may be areas that the user needs to focus on, these areas require more refined measurements and higher error requirements. For example, when scanning crack areas in the pipeline, since the size of the cracks must be accurately determined in order to take corresponding treatment measures based on the crack size, these areas require more refined scanning. Therefore, after the AR device captures the real-life image, it can determine whether the area currently being scanned by the 3D scanning device is a specific type of area based on the real-life image. For example, based on the matching relationship between the reconstructed 3D model and the real-life image, it can determine which area of ​​the real-life image the current 3D model corresponds to. This area is the area currently being scanned. The trained model can then be used to identify the area to determine whether it is a specific type of area. If so, the 3D scanning device can be notified to switch the scanning mode to a high-precision scanning mode to enable a more refined scan of the area.

[0099] S310: Display a preset image in the AR device, where the preset image includes: a first augmented reality image.

[0100] In some embodiments, the method can be executed by an AR device. Considering that the process of determining the matching relationship between the three-dimensional model and the target object in the real-scene image requires a large amount of calculation, some AR devices have weak processing performance, which may affect the real-time display of the first augmented reality image. Therefore, the AR device can detect whether there is a target device in the surrounding environment that can provide computing services. The target device can be a device with higher processing performance. After detecting the target device, the three-dimensional model and the real-scene image can be sent to the target device, and the matching relationship between the two can be determined by the target device. Based on the matching relationship, the three-dimensional model is superimposed on the real-scene image to obtain an augmented reality image, and then the first augmented reality image is received from the target device and displayed. By determining the matching relationship between the three-dimensional model and the real-scene image with the help of a device with better processing performance, the processing efficiency can be improved to ensure that the first augmented reality image is displayed in real time in the AR device.

[0101] In some scenarios, it may be necessary to scan certain target objects multiple times to determine their changing trends over time. For example, a patient's oral cavity can be scanned at regular intervals to determine the changing trends of their teeth, or certain pipes can be scanned at regular intervals to determine the changes in their corrosion positions or cracks over time. Therefore, in some embodiments, after reconstructing the three-dimensional model of the target object in real time based on the current scanned data, a historical three-dimensional model reconstructed based on the historical scan data of the target object can also be obtained. The historical three-dimensional model can then be compared with the currently reconstructed three-dimensional model to determine whether there are some risk areas, such as areas where cracks in the pipes increase sharply. These risk areas can then be identified in the first augmented reality image so that they can be displayed to the user through the AR device. At the same time, the user can also be warned.

[0102] When overlaying a 3D model of a target object onto a video stream of its real-life image, matching errors often cause the resulting augmented reality image to jitter. For example, if the target object is teeth, errors in the determined matching relationship for the current frame of the real-life image may cause the teeth in the 3D model to not align perfectly with those in the current frame. Consequently, when the 3D model is projected onto the current frame of the real-life image, the projected area and the teeth in the real-life image may not align. For example, the teeth in the 3D model may be offset to the left of the teeth in the real-life image. Furthermore, errors in the determined matching relationship for the next frame of the real-life image may cause the teeth in the 3D model to be offset to the right of the teeth in the real-life image. Consequently, when the user views the resulting augmented reality image, the position of the 3D model may suddenly jump from the left to the right, creating a jittering effect. Overall, the augmented reality video stream viewed by the user is jittery, severely impacting the overall presentation quality.

[0103] Based on this, in some embodiments, when superimposing a three-dimensional model onto a real-time video stream captured by an AR device, when determining the matching relationship between the three-dimensional model and the current frame real-scene image in the video stream, the matching relationship between the three-dimensional model and the one or more frames of real-scene images before the current frame real-scene image can be determined together. That is, the matching relationship between the three-dimensional model and the previous one or more frames of real-scene images is used to constrain and optimize the matching relationship between the three-dimensional model and the current frame real-scene image in the video stream, which is equivalent to smoothing the matching relationship between the three-dimensional model and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the first augmented reality image finally generated, and improving the display effect of the video stream of the first augmented reality image.

[0104] For example, after obtaining the currently reconstructed three-dimensional model and the current frame real-scene image from the video stream captured by the AR device, the initial matching relationship between the three-dimensional model and the current frame real-scene image can be determined, and the initial matching relationship can be optimized based on the matching relationship between the three-dimensional model and the reference real-scene image to obtain a target matching relationship; wherein the reference real-scene image is one or more frames of images before the current frame real-scene image in the video stream; then, based on the target matching relationship, the three-dimensional model can be projected into the current frame real-scene image, the scanned area can be determined, and the identification processing can be carried out to obtain the first augmented reality image.

[0105] In some embodiments, when optimizing the initial matching relationship based on the matching relationship between the three-dimensional model and the reference real-scene image to obtain the target matching relationship, the constraint item can be first determined based on the deviation between the matching relationship between the three-dimensional model and the reference real-scene image and the initial matching relationship, and then the constraint item is used to optimize the initial matching relationship to obtain the target matching relationship.

[0106] In some embodiments, when optimizing the initial matching relationship based on the matching relationship between the 3D model and the reference image to obtain a target matching relationship, a first matching error can be constructed based on the matching relationship between the 3D model and the reference image, and a second matching error can be constructed based on the target matching relationship. The target matching relationship is then solved using the minimization of the first matching error and the second matching error as constraints. That is, the target matching relationship ultimately calculated minimizes the matching error between the 3D model and the reference real-scene image, and also minimizes the matching error between the 3D model and the real-scene image of the current frame.

[0107] As shown in FIG10 , the real scene image includes a current frame real scene image in a video stream captured from an AR device. The method may include the following steps:

[0108] S1002. Acquire a virtual image and a real-scene image of the current frame. The virtual image includes a three-dimensional model obtained by real-time three-dimensional reconstruction of a target object based on data collected by a three-dimensional scanning device, or pre-constructed image data including the target object.

[0109] In step S1002, a virtual image and a current frame of real-world image can be obtained from the video stream captured by the AR device. The virtual image can be a two-dimensional image, such as virtual patterns such as virtual glasses and a hat, or can be CBCT or CT data. The virtual image can also be a three-dimensional model, which can be a three-dimensional model reconstructed in real time or a pre-built three-dimensional model. The three-dimensional model can be a complete model or a partial model. The current frame of real-world image can be the real-world image captured by the AR device at the current moment.

[0110] The image data can be a model of teeth and roots. Usually, doctors can only see the teeth and gums in the mouth, but not the roots. If image data including the roots is pre-built, the doctor can wear AR glasses when examining the patient's mouth. AR glasses can capture real-life images of the patient's mouth in real time, and then superimpose the image data of the roots on each frame of the captured real-life image to obtain a second augmented reality image, which is then displayed on the display of the AR glasses, thereby helping the doctor clearly understand the condition of the root area of ​​each tooth.

[0111] S1004. Determine an initial matching relationship between the virtual image and the current frame real-scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real-scene image to obtain a target matching relationship; wherein the reference real-scene image is one or more frames of images before the current frame of the real-scene image in the video stream.

[0112] In step S1004, after acquiring the virtual image and the current frame real scene image, the matching relationship between the virtual image and the current frame real scene image can be determined based on the internal and external parameters of the camera of the AR device, hereinafter referred to as the initial matching relationship. Among them, the internal parameters of the camera are usually fixed, and the external parameters can be calculated based on the data collected by the inertial measurement unit IMU in the AR device and the data collected by the multiple sensors of the camera. The matching relationship is configured to unify the virtual image and the real scene image into the same coordinate system, that is, to align the two frames.

[0113] Since there may be matching errors in the process of matching the virtual image with each frame of real-scene image, and the matching errors of each frame of real-scene image are inconsistent, the final virtual image and the real-scene image cannot be accurately aligned, and the superimposed position of the virtual image in the real-scene image will also change, causing the video stream of the final generated augmented reality image to be jittery. In order to eliminate jitter, the initial matching relationship can be optimized using the matching relationship between one or more frames of images before the current frame of real-scene image (i.e., the reference real-scene image) and the virtual image to obtain a more accurate target matching relationship. Among them, the one or more frames of images can be the previous frame of the current frame of real-scene image, or the previous two or more frames of images, and can be flexibly set based on actual needs.

[0114] S1006. Superimpose the virtual image onto the current frame real scene image based on the target matching relationship to obtain a second augmented reality image, so as to display the second augmented reality image in the AR device.

[0115] In step S1006, after determining the target matching relationship, the virtual image and the current frame real scene image can be aligned based on the target matching relationship, that is, the two can be unified into the same coordinate system. Then, based on the alignment result, the virtual image is superimposed on the current frame real scene image to obtain a second augmented reality image, so that the second augmented reality image can be displayed on the AR device. In the scene where the virtual image is a two-dimensional image, the two-dimensional image and the current frame real scene image can be directly fused. In the scene where the virtual image is a three-dimensional model, the three-dimensional model can be projected onto the current frame real scene image, and the projection area can be marked.

[0116] For each frame of real-scene image captured by the AR device, the virtual image can be superimposed on the real-scene image using the above method. Thus, the matching relationship between each frame of real-scene image and the virtual image is constrained by the matching relationship between the previous one or more frames of real-scene image and the virtual image. This can reduce the matching error and eliminate the jitter of the final generated augmented reality image.

[0117] In some embodiments, when optimizing the initial matching relationship based on the matching relationship between the virtual image and the reference real-scene image to obtain the target matching relationship, a constraint item can be first determined based on the deviation between the matching relationship between the virtual image and the reference real-scene image and the initial matching relationship, and then the constraint item can be used to optimize the initial matching relationship to obtain the target matching relationship.

[0118] In some embodiments, when optimizing the initial matching relationship based on the matching relationship between the virtual image and the reference image to obtain a target matching relationship, a first matching error can be constructed based on the matching relationship between the virtual image and the reference image, and a second matching error can be constructed based on the target matching relationship. The target matching relationship is then solved using the minimization of the first matching error and the second matching error as constraints. That is, the target matching relationship ultimately calculated will minimize the matching error between the virtual image and the reference real-world image, and also minimize the matching error between the virtual image and the real-world image of the current frame.

[0119] In some embodiments, the method can be configured as a scenario in which a three-dimensional scanning device is used to perform three-dimensional scanning on a target object, and the virtual image is a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device during the scanning of the target object using the three-dimensional scanning device, or pre-constructed image data including the target object.

[0120] In some embodiments, when the three-dimensional model is superimposed on the current frame real scene image, the three-dimensional model can be projected into the current frame real scene image based on the target matching relationship to obtain the projection area of ​​the three-dimensional model in the current frame real scene image, and the projection area is the scanned area in the current frame real scene image. The scanned area can then be identified to obtain a second augmented reality image. In this scenario, the matching relationship is the correspondence between the three-dimensional points in the three-dimensional model and the pixel points in the real scene image. The matching relationship can be determined based on the internal parameters and external parameters of the AR device (the relative position relationship between the AR device and the three-dimensional scanning device). Among them, the identification processing method is the same as the method of identifying the scanned area in the real scene image in reference Figure 3, and this application will not repeat the description.

[0121] During the real-time scanning of the target object, the real-time reconstructed 3D model is projected onto the real-scene image to determine the scanned area, and the scanned area is marked to obtain a second augmented reality image. The second augmented reality image can then be displayed on the display screen of the AR device, so that the user can see the real-time status of the target object and the 3D scanning device, as well as the real-time reconstructed 3D model. Therefore, the user can determine how to adjust the position of the 3D scanning device and determine the scanning method simply by viewing the content displayed on the AR device display screen, without having to switch the line of sight back and forth during the scanning process. The operation is very convenient and improves the user experience during the scanning process.

[0122] In some embodiments, the scanned area can also be divided into multiple sub-areas, which can be divided equally or unequally. The scanning quality of each sub-area can then be determined and identified in the second augmented reality image so that the user can intuitively understand the scanning status of the currently scanned area.

[0123] S1008: Displaying a preset image on the AR device.

[0124] In some embodiments, the preset image includes: a first augmented reality image, and / or a second augmented reality image, and / or an image derived from a fusion of the first augmented reality image and the second augmented reality image. The user can selectively switch the image displayed on the AR device as needed, or the AR device may initially display any of the preset images by default.

[0125] In some embodiments, to facilitate user scanning, before displaying a preset image in real time via an AR device, scanning trajectory data matching the target object can be acquired. Based on the acquired scanning trajectory data, a scanning trajectory line can then be drawn within the preset image. The scanning trajectory line is configured to guide the user in performing a three-dimensional scan of the target object using a 3D scanning device. For example, for each type of target object, an optimal scanning trajectory line for that type of target object can be pre-designed, and data related to the scanning trajectory, i.e., scanning trajectory data, can be stored. This scanning trajectory data is configured to indicate the position and scanning direction of the scanning trajectory line. By drawing the scanning trajectory line within the preset image, the preset image ultimately displayed to the user also includes the scanning trajectory line, allowing the user to move the 3D scanning device along the scanning trajectory line in the preset image to scan the target object. This approach can significantly improve scanning efficiency, facilitate user operation, and enable accurate scanning of previously unscanned target objects, resulting in superior scanning results. As shown in FIG4 , taking the oral cavity as an example, a scanning trajectory line can be drawn within the preset image to guide the user in scanning the oral cavity.

[0126] Considering that while the user is scanning the target object in real time, there may be other users who also want to know the real-time scanning status of the target object. For example, taking oral scanning as an example, when the doctor is scanning the patient's teeth with an oral scanning device, the patient may also want to know the condition of his or her own oral cavity at the same time. Therefore, in some embodiments, the AR device may be a first AR device, and after generating a preset image, the first AR device may send the preset image to the second AR device so that the second AR device also displays the preset image. For example, the first AR device may be AR glasses worn by a doctor, and the doctor can perform a real-time scan of the patient's oral cavity based on the preset image displayed by the AR glasses. At the same time, the patient can also wear AR glasses, that is, the second AR device, and the patient's AR glasses can also obtain the preset image and display it to the patient. Thus, during the scanning process, both the doctor and the patient can see the condition of the oral cavity and the current scanning status.

[0127] In some embodiments, considering that during the scanning of the target object, the user wearing the first AR device and the user wearing the second AR device may need to communicate and discuss a specific part of the target object. In order to facilitate one user to know which area of ​​the target object the other user is currently introducing, a user instruction input by the user through the first AR device or the second AR device can be received. For example, the user can use the control component of the AR device, or the user can input the user instruction through voice or gesture. After receiving the user instruction, the intention of the user instruction can be analyzed to determine the target area indicated by the user instruction, and the target area can be identified from the preset image. The target area is marked, for example, the target area is framed to display the preset image after marking in the first AR device and the second AR device.

[0128] In some embodiments, the target object may be an oral cavity, and the target area may be a tooth. In order to accurately identify the tooth that the doctor or patient wants to discuss, when identifying the target area indicated by the user instruction from the preset image and identifying the target area, the preset image can be semantically segmented first to obtain the various teeth in the patient's mouth, and then the tooth indicated by the user instruction can be identified from the segmented teeth and the tooth can be identified.

[0129] For example, as shown in Figure 7, during a scan, a doctor may need to explain the patient's oral condition, such as explaining that a certain tooth has caries and requires treatment, or that two teeth have a misaligned occlusion and require correction. To help the patient understand the specific tooth or area of ​​the mouth the doctor is referring to, the doctor can enter a user command through the AR glasses they are wearing before introducing the patient. This command indicates the target area the doctor wants to introduce. For example, the doctor can enter a command through voice or gesture, pointing to the target area they want to introduce. For example, the doctor can enter "Nth tooth from the left to the right in the top row" or "a certain tooth number" by voice. The AR glasses will then recognize the voice command, identify the tooth indicated by the voice command from a preset image, mark the tooth, and then display the marked preset image on both the doctor's AR glasses and the patient's AR glasses. Both the doctor and the patient can then see the marked tooth in the preset image displayed on their AR glasses, knowing which tooth the doctor is introducing. The same principle applies to patients. When a patient wants to consult a doctor about a certain tooth, he or she can also input user instructions indicating the tooth through the AR glasses they wear. The AR glasses can recognize the tooth from the preset image and mark it, and then display the marked preset image in the AR glasses of the doctor and the patient, so that the doctor also knows which tooth the patient is currently consulting about.

[0130] Generally speaking, during the scanning process, users may be more concerned about certain types of areas of the target object. Therefore, these areas can be marked in the second augmented reality image for easy viewing by the user. In order to allow users to intuitively view specific types of areas during the scanning process, in some embodiments, after the AR device displays the second augmented reality image, the second augmented reality image can also be recognized. For example, a pre-trained image recognition model can be used to recognize the second augmented reality image to determine whether the specific type of area exists in the second augmented reality image. If so, the specific type of area is marked.

[0131] In some embodiments, the method for determining the target matching relationship based on the current frame real-scene image is similar to the method for determining the matching relationship based on the real-scene image in Figure 3, and this application will not repeat the description. In some embodiments, this method can be performed by an AR device. Considering that the process of determining the matching relationship between the target object in the three-dimensional model and the real-scene image and aligning the two and then superimposing them is computationally intensive, some AR devices have weak processing performance, which may affect the real-time display of the augmented reality image. Therefore, the AR device can detect whether there is a target device in the surrounding environment that can provide computing services. The target device can be a device with higher processing performance. After detecting the target device, the three-dimensional model and the real-scene image can be sent to the target device, and the matching relationship between the two can be determined by the target device. Based on the matching relationship, the three-dimensional model is superimposed on the real-scene image to obtain a second augmented reality image, and then the second augmented reality image is received from the target device and displayed. By using a device with better processing performance to achieve alignment and superposition of the three-dimensional model and the real-scene image, processing efficiency can be improved, ensuring that the second augmented reality image is displayed in real time on the AR device.

[0132] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0133] Accordingly, an embodiment of the present disclosure further provides an image processing device 80, as shown in FIG11 , comprising: a three-dimensional model acquisition module 81 configured to acquire a three-dimensional model of a target object in real time during a process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device;

[0134] A real scene image acquisition module 82 is configured to acquire a real scene image of a target object captured by an AR device in real time;

[0135] a projection module 83 configured to determine a matching relationship between the three-dimensional model and the target object in the real image, and project the three-dimensional model into the real image based on the matching relationship to determine a scanned area in the real image;

[0136] The identification module 84 identifies the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through the AR device.

[0137] The specific steps of the image processing method executed by the above-mentioned device can be referred to the description in the above-mentioned method embodiment, which will not be repeated here.

[0138] The image processing device 80 further includes an image processing device 120. Accordingly, the embodiment of the present disclosure further provides an image processing device 120, as shown in FIG12 . The real scene image includes a current frame real scene image in a video stream collected from an AR device. The device includes:

[0139] The acquisition module 1201 is configured to acquire a virtual image and a current frame real scene image; the virtual image includes a 3D model obtained by real-time 3D reconstruction of a target object based on data collected by a 3D scanning device, or pre-constructed image data including the target object;

[0140] Matching module 1202 is configured to determine an initial matching relationship between the virtual image and the current frame of the real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images preceding the current frame of the real scene image in the video stream;

[0141] The superposition module 1203 is configured to superimpose the virtual image onto the current frame real scene image based on the target matching relationship to obtain a second augmented reality image, so as to display the second augmented reality image on the AR device;

[0142] The display module 1204 is configured to display a preset image in the AR device, where the preset image includes: a first augmented reality image, and / or a second augmented reality image, and / or an image obtained by fusing the first augmented reality image and the second augmented reality image.

[0143] The specific steps of the image processing method executed by the above-mentioned device can be referred to the description in the above-mentioned method embodiment, which will not be repeated here.

[0144] Furthermore, an embodiment of the present disclosure also provides a device, as shown in Figure 13, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 for execution by the processor 91. When the processor 91 executes the computer instructions, it implements any method in the above embodiments.

[0145] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, which implements the method of any of the aforementioned embodiments when the program is executed by a processor.

[0146] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be configured to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0147] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment of the embodiments of the present disclosure or certain parts of the embodiments.

[0148] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, and the modules described as separate components may or may not be physically separated. When implementing the embodiment of the present disclosure, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0150] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the embodiment of the present disclosure. Industrial Applicability

[0151] When a user uses a 3D scanning device to scan a target object, a 3D model of the target object can be obtained by real-time 3D reconstruction of the target object based on the data collected by the 3D scanning device. At the same time, a real-life image of the target object can be captured by a camera on the AR device, and a matching relationship between the 3D model and the target object in the real-life image can be determined. Based on the matching relationship, the 3D model is projected onto the real-life image to determine the scanned area in the real-life image, and the scanned area in the real-life image is identified to obtain a first augmented reality image. The first augmented reality image can then be displayed on the display screen of the AR device. By viewing the first augmented reality image, the user can see the real-time status of the target object and the 3D scanning device, as well as the current scanning status of the target object. Therefore, the user can determine how to adjust the position of the 3D scanning device and determine the scanning method simply by viewing the content displayed on the AR device display screen, without having to switch the line of sight back and forth during the scanning process. The operation is very convenient, improving the user experience during the scanning process, and has strong industrial practicality.

Claims

1. An image processing method, wherein: The method comprises: In the process of scanning the target object by using the three-dimensional scanning device, a three-dimensional model of the target object is obtained in real time, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device; Acquire the real-life image of the target object captured by the AR device in real time; Determining a matching relationship between the three-dimensional model and a target object in the real scene image, and projecting the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image; Marking the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through the AR device; A preset image is displayed in the AR device, where the preset image includes: the first augmented reality image.

2. The method according to claim 1, wherein: The real scene image includes a current frame real scene image in a video stream collected from the AR device, and the method further includes: Acquire a virtual image and the real scene image of the current frame; the virtual image includes the three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on the data collected by the three-dimensional scanning device, or pre-constructed image data including the target object; Determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream; Based on the target matching relationship, the virtual image is superimposed on the real scene image of the current frame to obtain a second augmented reality image, so as to display the second augmented reality image in the AR device; A preset image is displayed in the AR device, where the preset image includes: the first augmented reality image, and / or the second augmented reality image, and / or an image obtained by fusion of the first augmented reality image and the second augmented reality image.

3. The method according to claim 1 or 2, wherein: Before displaying the preset image in the AR device, the method further includes: Acquire scanning trajectory data matching the target object; A scanning trajectory line is drawn in the preset image based on the scanning trajectory data, wherein the scanning trajectory line is configured to guide a user to use the three-dimensional scanning device to perform a three-dimensional scan on the target object.

4. The method according to claim 1 or 2, wherein: The method is performed by the AR device, and the method further includes: Detect whether there is a target device in the surrounding environment that can provide computing services; After detecting the target device, the three-dimensional model and the real scene image are sent to the target device so that the target device determines the matching relationship, and based on the matching relationship, the three-dimensional model is projected into the real scene image to determine the scanned area in the real scene image, and the scanned area in the real scene image is marked to obtain a preset image; The preset image is received from the target device and displayed.

5. The method according to claim 1 or 2, wherein: Before displaying the preset image in the AR device, the method further includes: Acquire a historical three-dimensional model of the target object, wherein the historical three-dimensional model is reconstructed based on historical scanning data of the target object; Comparing the historical three-dimensional model with the three-dimensional model reconstructed in real time to determine the risk area in the target object; The risk area is identified in the real scene image.

6. The method according to claim 1 or 2, wherein: Before displaying the preset image in the AR device, the method further includes: A specific type of region is identified from the preset image, and a marking process is performed on the specific type of region.

7. The method according to claim 6, wherein: The target object is the oral cavity, and the specific type of area includes an oral disease area; the target object is a pipe, and the specific type of area includes an area where cracks or corrosion occur.

8. The method according to claim 1 or 2, wherein: The AR device is a first AR device, and the method further includes: The preset image is sent to the second AR device so that the second AR device displays the preset image.

9. The method according to claim 8, wherein: The method further comprises: receiving a user instruction input by a user through the first AR device or the second AR device; A target area indicated by the user instruction is identified from the preset image, and a marking process is performed on the target area, so that the preset image after the marking process is displayed in the first AR device and the second AR device.

10. The method according to claim 9, wherein: The target object is an oral cavity, the target area is a tooth, and the target area indicated by the user instruction is identified from the preset image, and the target area is marked, including: Performing semantic segmentation processing on the preset image to obtain each tooth in the oral cavity; The tooth indicated by the user instruction is identified from the teeth obtained by segmentation, and the tooth is marked.

11. The method according to claim 1, wherein: The method further comprises: dividing the scanned area into a plurality of sub-areas; The scanning quality of each sub-region in the scanned region is determined, and the scanning quality of each sub-region is identified in the first augmented reality image.

12. The method according to claim 1, wherein: The target object is an oral cavity, and determining a matching relationship between the three-dimensional model and the target object in the real scene image includes: Performing semantic segmentation processing on the three-dimensional model to obtain multiple tooth models; Performing semantic segmentation processing on the target object in the real scene image to obtain multiple tooth regions; Determining a correspondence between the plurality of tooth models and the plurality of tooth regions based on the tooth numbers; A matching relationship between the three-dimensional model and the target object in the real scene image is determined based on the corresponding relationship.

13. The method according to claim 12, wherein: Determining a matching relationship between the three-dimensional model and a target object in the real scene image based on the corresponding relationship includes: Determining a tooth model and a tooth region corresponding to a same tooth from among the plurality of tooth models and the plurality of tooth regions based on the corresponding relationship; Extracting a first boundary line between teeth and gums from the tooth model, and extracting a second boundary line between teeth and gums from the tooth region; Projecting the first dividing line into the real scene image to obtain a projection line, and constructing a reprojection error between the projection line and the second dividing line; The matching relationship between the three-dimensional model and the target object in the real scene image is determined at least with the minimization of the reprojection error as a constraint condition.

14. The method according to claim 1, wherein: The scanning mode of the three-dimensional scanning device includes a high-precision scanning mode and a low-precision scanning mode, and the method further includes: In the case where it is determined based on the real scene image that the area currently scanned by the three-dimensional scanning device is an area of ​​a specific type, the three-dimensional scanning device is notified to switch the scanning mode to a high-precision scanning mode.

15. The method according to claim 2, wherein: The initial matching relationship is optimized based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship, including: Determine a constraint item based on the deviation between the matching relationship between the virtual image and the reference real scene image and the initial matching relationship, and optimize the initial matching relationship based on the constraint item to obtain the target matching relationship; or A first matching error is constructed based on the matching relationship between the virtual image and the reference real scene image, and a second matching error is constructed based on the target matching relationship; and the target matching relationship is obtained by solving the minimization of the first matching error and the second matching error as a constraint condition.

16. An image processing device, wherein: The device comprises: A three-dimensional model acquisition module is configured to acquire a three-dimensional model of the target object in real time during a process of scanning the target object using a three-dimensional scanning device, wherein the three-dimensional model is obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device; A real scene image acquisition module is configured to acquire a real scene image of a target object captured by an AR device in real time; a projection module, configured to determine a matching relationship between the three-dimensional model and a target object in the real scene image, and project the three-dimensional model into the real scene image based on the matching relationship to determine a scanned area in the real scene image; an identification module, configured to perform identification processing on the scanned area in the real scene image to obtain a first augmented reality image, so as to display the first augmented reality image in real time through the AR device; A display module is configured to display a preset image in the AR device, and the preset image includes: the first augmented reality image.

17. The device according to claim 16, wherein: The real scene image includes a current frame real scene image in a video stream collected from the AR device, and the apparatus further includes: an acquisition module, configured to acquire a virtual image and the real scene image of the current frame; the virtual image includes the three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on the data collected by the three-dimensional scanning device, or pre-constructed image data including the target object; A matching module is configured to determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream; A superposition module is configured to superimpose the virtual image onto the current frame real scene image based on the target matching relationship to obtain a second augmented reality image, so as to display the second augmented reality image in the AR device; The display module is also configured to display a preset image in the AR device, and the preset image includes: the first augmented reality image, and / or the second augmented reality image, and / or an image obtained by fusion of the first augmented reality image and the second augmented reality image.

18. An electronic device, wherein: The electronic device includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method according to any one of claims 1 to 15 is implemented.

19. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images