Image processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, equipment and a storage medium. In the process of superposing the virtual image to the real-time video stream collected by the AR equipment, when the matching relationship between the virtual image and the current frame of real-scene image in the video stream is determined, the real-scene image can be superposed according to the matching relationship between the virtual image and the previous one or more frames of real-scene images; the matching relationship between the virtual image and the current frame of real-scene image in the video stream is constrained and optimized, equivalently, the matching relationship between the virtual image and different frames of real-scene images is smoothed, so that the problems of jitter and jump in the video stream of the finally generated augmented reality image can be eliminated; and the display effect of the video stream of the augmented reality image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] AR (Augmented Reality) technology can enhance the real world by superimposing virtual information onto the real world. In some application scenarios, some virtual images need to be superimposed onto each frame of real-scene images of the real-time video stream collected by the AR device to obtain an augmented reality image video stream, so that users can obtain an augmented reality image video stream with richer and more comprehensive information. At present, when performing superposition processing, the superposition of each frame of real-scene images and virtual images is processed independently. For example, for the current frame of real-scene images, the matching relationship between the virtual image and the target object in the real-scene image can be determined based on the internal and external parameters of the camera of the AR device, and the two are aligned based on the matching relationship, and the virtual image is superimposed onto the real-scene image. For the next frame of real-scene images, a similar method is used to complete the superposition of the two. Since there is a certain matching error when determining the matching relationship between the virtual image and each frame of the real-scene image, the video stream of the augmented reality image finally displayed will have jump and jitter problems. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0005] Obtain a virtual image and obtain the current frame of real-scene image from the video stream captured by the AR device;

[0006] Determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream;

[0007] Based on the target matching relationship, the virtual image is superimposed on the real scene image of the current frame to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

[0008] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0009] An acquisition module is used to acquire a virtual image and obtain a real scene image of the current frame from a video stream captured by an AR device;

[0010] A matching module, used to determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream;

[0011] The overlay module is used to overlay the virtual image onto the real scene image of the current frame based on the target matching relationship to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

[0012] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein when the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0013] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0014] In the disclosed embodiment, in the process of superimposing the virtual image onto the real-time video stream of the target object, when determining the matching relationship between the virtual image and the current frame real-scene image in the video stream, the matching relationship between the virtual image and the one or more frames of real-scene images before the current frame real-scene image can be jointly determined. That is, the matching relationship between the virtual image and the previous one or more frames of real-scene images is used to constrain and optimize the matching relationship between the virtual image and the current frame real-scene image in the video stream, which is equivalent to smoothing the matching relationship between the virtual image and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the augmented reality image finally generated, and improving the display effect of the video stream of the augmented reality image.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0017] Figure 1 It is a schematic diagram of an embodiment of the present disclosure in which a three-dimensional tooth model is superimposed on a real-scene image video stream with a jump problem.

[0018] Figure 2It is a schematic diagram of determining a matching relationship in an embodiment of the present disclosure.

[0019] Figure 3 It is a schematic diagram of a three-dimensional scanning method according to an embodiment of the present disclosure.

[0020] Figure 4 It is a schematic diagram of identifying a scanning trajectory line in an augmented reality image according to an embodiment of the present disclosure.

[0021] Figure 5 It is a schematic diagram of identifying scanning quality in an augmented reality image according to an embodiment of the present disclosure.

[0022] Figure 6 It is a schematic diagram of identifying a specific type of area in an augmented reality image according to an embodiment of the present disclosure.

[0023] Figure 7 It is a schematic diagram of an embodiment of the present disclosure, which identifies the tooth currently under discussion in an augmented reality image displayed by AR glasses worn by a doctor and a patient.

[0024] Figure 8 It is a schematic diagram of the logical structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0025] Fig. 9 It is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0027] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0028] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0030] AR technology can enhance the real world by superimposing virtual information onto the real world. In some application scenarios, some virtual images need to be superimposed onto each frame of real-scene images in the real-time video stream collected by the AR device to obtain an augmented reality image video stream, so that users can obtain an augmented reality image video stream with richer and more comprehensive information.

[0031] For example, for some scenes, users need to add some virtual information to the real scene to make the real scene more realistic. For example, when a user takes a selfie, virtual glasses or a virtual hat are automatically added to the user's face image.

[0032] For example, in a scenario where a user performs a real-time three-dimensional scanning of a target object, in order to facilitate the user to understand the current scanning situation, the data collected by the three-dimensional scanning device can be used for real-time three-dimensional reconstruction to obtain a three-dimensional model of the target object. At the same time, the AR device can be used to collect a video stream of the target object in real time, and then the real-time reconstructed three-dimensional model is superimposed on each frame of the real-scene image of the video stream to obtain an augmented reality image video stream, so that the user can know the real-time scanning situation of the target object by viewing the augmented reality image video stream, so as to guide the user to perform a three-dimensional scanning of the target object. Among them, the three-dimensional scanning device can be a handheld or fixed scanner such as an oral scanner, a facial scanner, an ear scanner, an industrial scanner, a professional scanner, etc., which can realize the three-dimensional reconstruction of objects or scenes such as teeth, human faces, ears, human bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical instruments, buildings, etc.

[0033] For another example, in certain scenes, users can only see the surface of the target object, but cannot see the internal situation of the target object. In order to allow users to see the internal situation of the target object when viewing the target object, a 3D model of the target object can be reconstructed in advance, and then the video stream of the target object can be collected in real time using an AR device, and the pre-reconstructed 3D model can be superimposed on each frame of the real scene image of the video stream to obtain an augmented reality image video stream, that is, the user can know the internal situation of the target object by viewing the augmented reality image video stream. Taking the oral cavity as an example, doctors can usually only see the teeth and gums in the oral cavity, but cannot see the root part. If a 3D model of the root part is pre-constructed, when the doctor views the patient's oral cavity, the doctor can wear AR glasses, which can collect the real scene image of the patient's oral cavity in real time, and then superimpose the 3D model of the root on each frame of the collected real scene image to obtain an augmented reality image and then display it on the display screen of the AR glasses, so that the doctor can clearly understand the situation of the root part of each tooth.

[0034] For the above-mentioned scenarios or similar scenarios, since the virtual image needs to be superimposed on each frame of the real-scene image of the real-time video stream collected by the AR device, the superposition of each frame of the real-scene image and the virtual image is currently processed independently during the superposition process. For example, for the current frame of the real-scene image, the matching relationship between the virtual image and the real-scene image can be determined based on the internal and external parameters of the camera of the AR device, and the two can be aligned based on the matching relationship, and the virtual image can be superimposed on the real-scene image. For the next frame of the real-scene image, a similar method is used to complete the superposition of the two. Since there is a certain matching error when determining the matching relationship between the virtual image and each frame of the real-scene image, the augmented reality image video stream that is finally displayed will have jump and jitter problems.

[0035] For example, Figure 1 As shown, taking the target object as the oral cavity as an example, when performing a real-time three-dimensional scan of the oral cavity, in order to facilitate the user to understand the current scanning situation, the three-dimensional model of the oral cavity reconstructed in real time based on the data collected by the three-dimensional scanning device can be superimposed on the real-scene image of the oral cavity, where superposition refers to projecting the three-dimensional model into the real-scene image, determining the scanned area, and marking the scanned area, so that the user can understand which areas are currently scanned areas.

[0036] In the process of superimposing the three-dimensional model of the oral cavity onto the real-life image video stream of the oral cavity, for example, for the current frame of the real-life image, the teeth in the three-dimensional model and the teeth in the current frame of the real-life image may not be completely aligned due to errors in the determined matching relationship, so that when the three-dimensional model is projected onto the current frame of the real-life image, the teeth in the projection area and the real-life image cannot be aligned, for example, the teeth in the three-dimensional model are offset to the left by a certain distance compared to the teeth in the real-life image. For the next frame of the real-life image, the teeth in the three-dimensional model may be offset to the right by a certain distance compared to the teeth in the real-life image due to errors in the determined matching relationship. Therefore, when the user sees the two frames of augmented reality images generated in the end, the position of the three-dimensional model may suddenly jump from the left position of the image to the right position, that is, there is a jump problem. From the overall display effect, the video stream of the augmented reality image seen by the user has been shaking, which seriously affects the display effect of the augmented reality image video stream.

[0037] The scenario of superimposing other virtual images onto the real-life image video stream is similar. For example, superimposing virtual glasses on a face image collected in real time may cause the virtual glasses to be always to the left or right of the face due to matching errors, causing the video stream of the augmented reality image seen by the user to be constantly jittery, seriously affecting the display effect of the augmented reality image video stream.

[0038] Based on this, the present disclosure provides an image processing method, such as Figure 2 As shown, in the process of superimposing the virtual image on the real-time video stream captured by the AR device, when determining the matching relationship between the virtual image and the current frame real-scene image in the video stream, the matching relationship between the virtual image and the one or more frames of real-scene images before the current frame real-scene image can be jointly determined, that is, the matching relationship between the virtual image and the previous one or more frames of real-scene images is constrained and optimized through the matching relationship between the virtual image and the current frame real-scene image in the video stream, which is equivalent to smoothing the matching relationship between the virtual image and different frames of real-scene images, thereby eliminating the jitter and jump problems in the video stream of the finally generated augmented reality image, and improving the display effect of the video stream of the augmented reality image.

[0039] In some scenarios, the method of the embodiments of the present disclosure can be executed by an AR device. For example, the AR device can obtain a virtual image. At the same time, the AR device can collect video streams in real time through a camera, determine the matching relationship between the virtual image and each frame of the real scene image in the video stream, and then align and superimpose the two based on the matching relationship to obtain and display an augmented reality image.

[0040] In some scenarios, the method of the disclosed embodiment may also be executed by other devices connected to the AR device, such as laptops, tablet computers, cloud servers, and other devices with good processing performance. These devices can obtain the virtual image, and at the same time, can obtain the video stream collected by the AR device in real time, determine the matching relationship between the virtual image and each frame of the real scene image in the video stream, and then align and superimpose the two based on the matching relationship to obtain an augmented reality image, which is then sent to the AR device, and the AR device displays the augmented reality image.

[0041] In some scenarios, the method of the embodiment of the present disclosure may also be that some steps are executed by the AR device and some steps are executed by other devices. For example, the step of determining the matching relationship is executed by other devices, and the step of superimposing and displaying the augmented reality image is executed by the AR device. The setting of the execution subject can be flexibly set based on actual needs, and the embodiment of the present disclosure does not limit it.

[0042] The virtual image in the disclosed embodiment may have various information that needs to be superimposed on the real scene, for example, it may be a two-dimensional image, a three-dimensional model, etc. For example, it may be an oral cavity, a human organ, etc., and the embodiment of the present application does not limit this.

[0043] The AR device in the embodiments of the present disclosure may be various types of AR devices, such as AR glasses, AR display screens, etc. The AR device includes a camera for collecting real-life images. The AR device includes a display screen for displaying augmented reality images. The AR device may also include various sensors for determining the current position of the user's line of sight.

[0044] like Figure 3 As shown, the method may include the following steps:

[0045] S302, obtaining a virtual image, and obtaining a current frame real scene image from a video stream captured by an AR device;

[0046] In step S302, a virtual image may be obtained, and a real-scene image of the current frame may be obtained from a video stream captured by an AR device. The virtual image may be a two-dimensional image, such as virtual patterns such as virtual glasses and hats. The virtual image may also be a three-dimensional model, which may be a three-dimensional model reconstructed in real time or a pre-built three-dimensional model, which may be a complete model or a partial model. The real-scene image of the current frame may be a real-scene image captured by the AR device at the current moment.

[0047] S304, determining an initial matching relationship between the virtual image and the current frame real scene image, and optimizing the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream;

[0048] In step S304, after acquiring the virtual image and the real scene image of the current frame, the matching relationship between the virtual image and the real scene image of the current frame can be determined based on the internal and external parameters of the camera of the AR device, which is referred to as the initial matching relationship below. Among them, the internal parameters of the camera are usually fixed, and the external parameters can be calculated based on the data collected by the inertial measurement unit IMU in the AR device and the data collected by the multi-sensor of the camera. The matching relationship is used to unify the virtual image and the real scene image into the same coordinate system, that is, to align the two frames.

[0049] Since there may be matching errors in the process of matching the virtual image with each frame of real-scene image, and the matching errors of each frame of real-scene image are inconsistent, the final virtual image and the real-scene image cannot be accurately aligned, and the superimposed position of the virtual image in the real-scene image will also change, causing the video stream of the final generated augmented reality image to have jitter. In order to eliminate jitter, the initial matching relationship can be optimized using the matching relationship between one or more frames of images before the current frame of real-scene image (i.e., the reference real-scene image) and the virtual image to obtain a more accurate target matching relationship. Among them, the one or more frames of images can be the previous frame of the current frame of real-scene image, or the previous two or more frames of images, which can be flexibly set based on actual needs.

[0050] S306. Superimpose the virtual image onto the real scene image of the current frame based on the target matching relationship to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

[0051] In step S306, after determining the target matching relationship, the virtual image and the current frame real scene image can be aligned based on the target matching relationship, that is, the two are unified into the same coordinate system, and then the virtual image is superimposed on the current frame real scene image based on the alignment result to obtain an augmented reality image, so as to display the augmented reality image in the AR device. In the scene where the virtual image is a two-dimensional image, the two-dimensional image and the current frame real scene image can be directly fused, and in the scene where the virtual image is a three-dimensional model, the three-dimensional model can be projected onto the current frame real scene image, and the projection area can be marked.

[0052] For each frame of real-scene image captured by the AR device, the virtual image can be superimposed on the real-scene image using the above method, so that the matching relationship between each frame of real-scene image and the virtual image is constrained by the matching relationship between the previous one or more frames of real-scene image and the virtual image, which can reduce the matching error and eliminate the jitter phenomenon of the finally generated augmented reality image.

[0053] In some embodiments, when optimizing the initial matching relationship based on the matching relationship between the virtual image and the reference real-scene image to obtain the target matching relationship, a constraint item can be first determined based on the deviation between the matching relationship between the virtual image and the reference real-scene image and the initial matching relationship, and then the constraint item is used to optimize the initial matching relationship to obtain the target matching relationship.

[0054] In some embodiments, when the initial matching relationship is optimized based on the matching relationship between the virtual image and the reference image to obtain the target matching relationship, a first matching error can be constructed based on the matching relationship between the virtual image and the reference image, and a second matching error can be constructed based on the target matching relationship, and then the target matching relationship is solved with the minimization of the first matching error and the second matching error as constraints. That is, the target matching relationship finally calculated will minimize the matching error between the virtual image and the reference real scene image, and also minimize the matching error between the virtual image and the real scene image of the current frame.

[0055] In some embodiments, the method can be used in a scene where a target object is three-dimensionally scanned using a three-dimensional scanning device, and the virtual image is a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device during the process of scanning the target object using the three-dimensional scanning device. When the three-dimensional model is superimposed on the current frame real-scene image, the three-dimensional model can be projected into the current frame real-scene image based on the target matching relationship to obtain the projection area of ​​the three-dimensional model in the current frame real-scene image, and the projection area is the scanned area in the current frame real-scene image. The scanned area can then be marked to obtain an augmented reality image. In this scenario, the matching relationship is the correspondence between the three-dimensional points in the three-dimensional model and the pixel points in the real-scene image, and the matching relationship can be determined based on the internal parameters and external parameters of the AR device (the relative position relationship between the AR device and the three-dimensional scanning device).

[0056] There are many ways to mark the area, as long as the user can distinguish which areas have been scanned. For example, the scanned area and the unscanned area can be rendered in different colors, or the scanned area can be framed, or the projection points when the three-dimensional model is projected to the area can be displayed in the scanned area, where the density of the projection points can indicate the density of the three-dimensional model corresponding to the area. After obtaining the augmented reality image after the marking process, the augmented reality image can be displayed in real time through the AR device, so that the user can understand the current real-time scanning status of the target object by viewing the augmented reality image.

[0057] During the real-time scanning of the target object, the real-time reconstructed 3D model is projected into the real-scene image to determine the scanned area, and the scanned area is marked to obtain an augmented reality image. The augmented reality image can then be displayed on the display screen of the AR device, so that the user can see the real-time status of the target object and the 3D scanning device, as well as the real-time reconstructed 3D model by viewing the augmented reality image. Therefore, the user can determine how to adjust the position of the 3D scanning device and determine the scanning method by simply viewing the content displayed on the display screen of the AR device, without having to switch the line of sight back and forth during the scanning process. The operation is very convenient, which improves the user experience during the scanning process.

[0058] Considering that for different types of target objects, there is often a better scanning trajectory during the scanning process. Scanning the target object based on the better scanning trajectory can not only perform a comprehensive scan of the target object, but also improve the scanning efficiency. Therefore, in some embodiments, in order to facilitate user scanning, before displaying the augmented reality image in the AR device, the scanning trajectory data matching the target object can be obtained first, and then the scanning trajectory line can be drawn in the augmented reality image based on the acquired scanning trajectory data, wherein the scanning trajectory line is used to guide the user to use the three-dimensional scanning device to perform a three-dimensional scan of the target object. For example, for each type of target object, the best scanning trajectory line for this type of target object can be pre-designed, and the data related to the scanning trajectory line, that is, the scanning trajectory line data, can be stored. The scanning trajectory line data is used to indicate the position information and direction information of the scanning trajectory line. By drawing the scanning trajectory line in the augmented reality image, the scanning trajectory line is also included in the augmented reality image finally displayed, so that the user can move the three-dimensional scanning device along the scanning trajectory line in the augmented reality image to scan the target object. In this way, the scanning efficiency can be greatly improved, the user operation is convenient, and the target object that the user has not scanned before can also be accurately operated to obtain better scanning results. Figure 4 As shown, taking the target object as the oral cavity as an example, a scanning trajectory line can be drawn in the augmented reality image to guide the user to scan the oral cavity.

[0059] In addition, when users are scanning the target object, the scanning quality is also an issue that users are more concerned about, such as whether there are problems with missed scanning in various areas of the target object, or whether the scanning density of various areas meets the standard, whether there are problems such as sparse three-dimensional points after reconstruction in some areas, and insufficient accuracy. In order to allow users to understand the scanning quality of each area that has been scanned in real time, in some embodiments, the scanned area can also be divided into multiple sub-areas, which can be divided equally or unequally, and then the scanning quality of each sub-area can be determined, and the scanning quality of each sub-area can be identified in the augmented reality image, so that the user can intuitively understand the scanning status of the current scanned area. For example, after reconstructing a three-dimensional model based on data collected by a three-dimensional scanning device, the matching relationship between the three-dimensional model and the target object in the real-life image can be determined, and then the number and density of three-dimensional points corresponding to each sub-area of ​​the target object in the real-life image can be determined based on the matching relationship, wherein the denser the three-dimensional points, the greater the scanning density of the area, and the better the scanning quality. In some scenes, the scanning quality of each sub-area can be identified by the depth of color, for example, the darker the color, the denser the three-dimensional points reconstructed after scanning the sub-area, and the better the scanning quality. Figure 5 As shown, the color depth can be used to indicate the scanning quality. Darker teeth indicate that the reconstructed 3D points are denser, and lighter teeth indicate that the reconstructed 3D points are sparser and need to be re-scanned. In some scenarios, some points can also be covered in each sub-area. The denser the covered points, the denser the 3D points reconstructed after scanning the sub-area, and the better the scanning quality. Of course, the specific identification method can be flexibly selected based on actual needs, as long as the user can intuitively understand the scanning quality of each sub-area of ​​the scanned area based on the final augmented reality image, so that the user can re-scan those sub-areas with poor scanning quality, thereby collecting more and more complete data.

[0060] Generally speaking, during the scanning process, some specific types of areas of the target object may be areas of concern to the user. Therefore, these areas can be marked in the augmented reality image for the user to view. For example, in some scenarios, the target object is the oral cavity, and the specific type of area may be the diseased area in the oral cavity, such as caries. Doctors usually pay more attention to these parts of the oral cavity, so these areas can be marked. In some scenarios, the target object may be a pipe, and the specific type of area may be an area with cracks in the pipe, or an area with rust and corrosion in the pipe.

[0061] In order to allow the user to intuitively view the specific type of area during the scanning process, in some embodiments, after the AR device displays the augmented reality image, the augmented reality image can also be recognized. For example, a pre-trained image recognition model can be used to recognize the augmented reality image to determine whether the specific type of area exists in the augmented reality image. If so, the specific type of area is marked, for example. Figure 6 As shown, the area can be framed or rendered in a specific color, etc. The specific identification method can be flexibly set based on actual needs.

[0062] Considering that when the user is scanning the target object in real time, there may be other users who also want to know the real-time scanning status of the target object. For example, taking oral scanning as an example, when the doctor is scanning the patient's teeth with an oral scanning device, the patient may also want to know the condition of his or her oral cavity. Therefore, in some embodiments, the AR device may be a first AR device, and after generating the augmented reality image, the first AR device may send the augmented reality image to the second AR device so that the second AR device also displays the augmented reality image. For example, the first AR device may be AR glasses worn by a doctor, and the doctor can perform a real-time scan of the patient's oral cavity based on the augmented reality image displayed by the AR glasses. At the same time, the patient can also wear AR glasses, that is, the second AR device, and the patient's AR glasses can also obtain the augmented reality image and display it to the patient. Thus, during the scanning process, both the doctor and the patient can see the condition of the oral cavity and the current scanning condition.

[0063] In some embodiments, considering that during the scanning of the target object, the user wearing the first AR device and the user wearing the second AR device may need to communicate and discuss a specific part of the target object. In order to facilitate one of the users to know which area of ​​the target object the other user is currently introducing, a user instruction input by the user through the first AR device or the second AR device can be received. For example, the user can use the control component of the AR device, or the user can input the user instruction through voice or gesture. After receiving the user instruction, the intention of the user instruction can be analyzed, the target area indicated by the user instruction can be determined, and the target area can be identified from the augmented reality image, and the target area can be marked, for example, the target area can be framed to display the augmented reality image after marking in the first AR device and the second AR device.

[0064] In some embodiments, the target object may be an oral cavity, and the target area may be a tooth. In order to accurately identify the tooth that the doctor or patient wants to discuss, when identifying the target area indicated by the user instruction from the augmented reality image and identifying the target area, the augmented reality image may be first semantically segmented to obtain the various teeth in the patient's mouth, and then the tooth indicated by the user instruction may be identified from the segmented teeth, and the tooth may be identified.

[0065] For example, Figure 7 As shown, during the scanning process, the doctor may need to introduce the patient's oral condition, for example, telling the patient that a certain tooth is caries and needs treatment, or that two teeth have a bad bite and need correction, etc. In order to let the patient know which tooth or area in the mouth the doctor is currently introducing, the doctor can enter a user instruction through the AR glasses he wears before introducing it to the patient. The user instruction can indicate the target area that the doctor wants to introduce. For example, the doctor can input instructions through voice or gestures to point to the target area he wants to introduce. For example, the doctor can voice input "the Nth tooth from left to right in the upper row" or "a certain tooth position number". At this time, the AR glasses can recognize the voice instruction, identify the tooth indicated by the voice instruction from the augmented reality image, and mark the tooth, and then display the augmented reality image after marking in the AR glasses worn by the doctor and the AR glasses worn by the patient. As a result, both the doctor and the patient can see the marked teeth through the augmented reality image displayed by the AR glasses they wear, and know which tooth the doctor is currently introducing. The same principle applies to patients. When a patient wants to consult a doctor about a tooth, he or she can also input user instructions for the tooth through the AR glasses they wear. The AR glasses can recognize the tooth from the augmented reality image and mark it. The marked augmented reality image is then displayed in the doctor's and patient's AR glasses, so that the doctor also knows which tooth the patient is currently consulting about.

[0066] When the reconstructed three-dimensional model is superimposed on the real-life image, accurately determining the matching relationship between the three-dimensional model and the target object in the real-life image is a prerequisite for obtaining an augmented reality image with better effects. In some embodiments, if the target object is an oral cavity, in order to obtain a more accurate matching relationship, the matching relationship can be constrained or corrected with the help of the correspondence between the teeth in the three-dimensional model and the teeth in the real-life image to obtain a more accurate matching relationship. For example, the three-dimensional model can be semantically segmented to obtain multiple tooth models, and the oral cavity area in the real-life image can be semantically segmented to obtain multiple tooth areas. Among them, when the three-dimensional model is semantically segmented, the gum area and each tooth in the three-dimensional model of the oral cavity can be identified to obtain a gum model and multiple tooth models. Similarly, when the real-life image is semantically segmented, the gum area and each tooth in the image can also be identified to obtain a gum area and multiple tooth areas.

[0067] Since the multiple tooth models correspond to the teeth in the real mouth, and the tooth regions also correspond to the teeth in the real mouth, the correspondence between the multiple tooth models and the multiple tooth regions can be determined based on the tooth numbers, and the two can be matched one by one. Then, the correspondence can be used as a constraint to determine the matching relationship between the three-dimensional model and the target object in the real scene image. Since the corresponding relationship between teeth is added as a constraint, the target matching relationship finally determined is also more accurate.

[0068] For example, after obtaining a tooth model and a tooth region corresponding to a tooth in the real world, an easily extractable feature line (e.g., upper edge line) or an easily identifiable feature point in the tooth, such as a center point or the highest point of a protrusion, can be determined from the tooth model and the tooth region, respectively. The corresponding relationship between these feature lines or feature points in the tooth model and the tooth region is established as a constraint to determine the matching relationship between the three-dimensional model and the target object in the real scene image.

[0069] In some embodiments, in order to determine the matching feature line or feature point from the tooth model and the tooth region as accurately as possible, the feature line may be the boundary line between the tooth and the gum. Since each tooth and the gum region can be identified during the semantic segmentation process, and the boundary line between the tooth and the gum is relatively clear and easy to extract, the tooth model and the tooth region corresponding to the same tooth can be determined from the three-dimensional model and the current frame real scene image, respectively, and then the boundary line between the tooth and the gum can be extracted from the two, respectively, and the corresponding relationship of the above boundary line is used as a constraint to determine the target matching relationship.

[0070] For example, when using the correspondence between teeth as a constraint to determine the target matching relationship, the tooth model and tooth area corresponding to the same tooth can be determined based on the correspondence between multiple tooth models and tooth areas, and then the boundary between the tooth and the gums can be extracted from the tooth model, hereinafter referred to as the first boundary line. At the same time, the boundary between the tooth and the gums can be extracted from the tooth area, hereinafter referred to as the second boundary line, wherein the first boundary line and the second boundary line correspond to the boundary between the tooth and the gums in the same real tooth, and then the first boundary line can be projected into the reference real scene image to obtain a projection line, and a reprojection error between the projection line and the second boundary line is constructed, and then the reprojection error and the matching relationship between the three-dimensional model and the reference real scene image are used to optimize the initial matching relationship to obtain a target matching relationship, wherein the target matching relationship minimizes the projection error.

[0071] Since the boundary between teeth and gums is relatively clear and easy to extract, the tooth model and tooth area corresponding to the same tooth can be determined from the three-dimensional model and the current frame real-scene image respectively, and then the boundary between teeth and gums can be extracted from the two respectively. The above boundary line is used to determine the reprojection error, and then the reprojection error is used as a constraint to solve the target matching relationship, so that the final target matching relationship is more accurate.

[0072] For example, suppose the 3D scanning device is at t i The three-dimensional model of the tooth scanned at each moment is in the coordinate system of the three-dimensional scanning device: AR devices are i The real scene image collected at each moment is The real-life image can be used to identify and segment the tooth position number, and the tooth region set corresponding to each tooth can be obtained. Where k represents the tooth position number.

[0073] Similarly, the tooth 3D model can also be segmented and identified to obtain a 3D mesh subset belonging to each tooth. in represents the mesh subset belonging to tooth position k in the tooth model (the three-dimensional model of each tooth). The boundary line of the tooth is the boundary between the tooth and the gum, recorded as It is also a part with special semantics.

[0074] Through the AR device camera internal and external parameter matrix K, T i , the 3D point P in the 3D tooth model can be projected onto the real-scene image plane captured by the camera to obtain the image coordinates of the projection point: u = KT i P. The internal parameter matrix K is only related to the optical system of AR glasses and is constant during the shooting process. The external parameter matrix T iis the term to be solved, which is used to describe the relative position relationship between the 3D scanning device and the camera, and accurately solve the external parameter matrix T i It's the key, KT i This is the aforementioned target matching relationship.

[0075] Solving the external parameter matrix requires finding the correspondence between the teeth in the 3D model and the teeth in the 2D image, which can be done with the help of the tooth boundary line. Find the correspondence between the tooth boundary line in the 3D model and the 2D image, and determine the reprojection error between the two. Then, the external parameter matrix can be solved with minimizing the reprojection error as a constraint, as shown in formula (1):

[0076]

[0077] in, is the pixel coordinate of the jth tooth-gingival boundary line in the image at time ti, is the pixel coordinate of the jth tooth-gingival boundary line in the image at time ti-1. is the three-dimensional coordinate of the jth tooth-gingival boundary line in the three-dimensional model at time ti (i.e., the tooth-tooth boundary line of a certain tooth model), is the 3D coordinate of the jth tooth and gum boundary line in the 3D model at time ti-1, K is the camera intrinsic parameter matrix, T i is the external parameter matrix of the camera at time ti, T i -1 is the external parameter matrix of the camera at time ti-1, ω 1 and ω 2 is the preset coefficient.

[0078] When determining the external parameter matrix, by introducing the reprojection error of the tooth-gingival boundary line as a constraint condition, the final target matching relationship can be made more accurate.

[0079] In some embodiments, the three-dimensional scanning device may include two scanning modes, namely, a high-precision scanning mode and a low-precision scanning mode. In the high-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is small, that is, the measurement error is small, while in the low-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is large, that is, the measurement error is large. Considering that certain areas of the target object may be areas that the user needs to focus on, these areas require more refined measurements and have higher requirements for errors. For example, when scanning a pipeline, for crack areas in the pipeline, since the size of the cracks must be accurately determined in order to take corresponding treatment measures based on the size of the cracks, these areas need to be scanned more finely. Therefore, after the AR device acquires the real-life image, it can determine whether the area currently scanned by the 3D scanning device is a specific type of area based on the real-life image. For example, based on the matching relationship between the reconstructed 3D model and the real-life image, it can determine which area of ​​the real-life image the current 3D model corresponds to. This area is the area currently being scanned. The trained model can then be used to identify the area to determine whether it is a specific type of area. If so, the 3D scanning device can be notified to switch the scanning mode to a high-precision scanning mode to enable a more refined scan of the area.

[0080] In some embodiments, the method can be performed by an AR device. Considering that the process of determining the matching relationship between the three-dimensional model and the target object in the real scene image and aligning the two and then superimposing them requires a large amount of calculation, the processing performance of some AR devices is relatively weak, which may affect the real-time display of the augmented reality image. Therefore,

[0081] The AR device can detect whether there is a target device in the surrounding environment that can provide computing services. The target device can be a device with higher processing performance. After detecting the target device, the three-dimensional model and the real scene image can be sent to the target device, and the matching relationship between the two can be determined by the target device. Based on the matching relationship, the three-dimensional model is superimposed on the real scene image to obtain an augmented reality image, and then the augmented reality image is received from the target device and displayed. By using a device with better processing performance to achieve alignment and superposition of the three-dimensional model and the real scene image, the processing efficiency can be improved, ensuring that the augmented reality image is displayed in real time in the AR device.

[0082] In some scenarios, for certain target objects, it may be necessary to scan them multiple times to determine their changing trends over time. For example, for the patient's mouth, it can be scanned at regular intervals to determine the changing trends of its teeth, or for certain pipes, it can be scanned at regular intervals to determine the changes in its corrosion position over time, or the changes in cracks over time, etc. Therefore, in some embodiments, after the three-dimensional model of the target object is reconstructed in real time based on the current scanned data, a historical three-dimensional model reconstructed based on the historical scan data of the target object can also be obtained, and then the historical three-dimensional model can be compared with the currently reconstructed three-dimensional model to determine whether there are some risk areas, such as areas where pipe cracks increase sharply, and then these risk areas can be marked in the real-life image so that they can be displayed to the user through the AR device, and at the same time, the user can also be warned.

[0083] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0084] Accordingly, the present disclosure also provides an image processing device, such as Figure 8 As shown, the device comprises:

[0085] An acquisition module 81 is used to acquire a virtual image and a real scene image of a current frame from a video stream captured by an AR device;

[0086] A matching module 82 is used to determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream;

[0087] The superposition module 83 is used to superimpose the virtual image into the current frame real scene image based on the target matching relationship to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

[0088] Among them, the specific steps of the above-mentioned device executing the image processing method can refer to the description in the above-mentioned method embodiment, and will not be repeated here.

[0089] Furthermore, the present disclosure also provides an electronic device, such as Fig. 9 As shown, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 for execution by the processor 91. When the processor 91 executes the computer instructions, the method described in any one of the above embodiments is implemented.

[0090] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.

[0093] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0094] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0095] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain a virtual image and obtain the current frame of real-scene image from the video stream captured by the AR device; Determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream; Based on the target matching relationship, the virtual image is superimposed on the real scene image of the current frame to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

2. The method according to claim 1, characterized in that The initial matching relationship is optimized based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship, including: Determine a constraint item based on the deviation between the matching relationship between the virtual image and the reference real scene image and the initial matching relationship, and optimize the initial matching relationship based on the constraint item to obtain the target matching relationship; or A first matching error is constructed based on the matching relationship between the virtual image and the reference real scene image, and a second matching error is constructed based on the target matching relationship; and the target matching relationship is obtained by solving the minimization of the first matching error and the second matching error as a constraint condition.

3. The method according to claim 1, characterized in that The virtual image is a three-dimensional model obtained by real-time three-dimensional reconstruction of the target object based on data collected by the three-dimensional scanning device during scanning of the target object by the three-dimensional scanning device, and the virtual image is superimposed on the current frame real scene image based on the target matching relationship to obtain an augmented reality image, including: The three-dimensional model is projected into the current frame real scene image based on the target matching relationship to determine the scanned area in the current frame real scene image, and the scanned area is marked to obtain an augmented reality image.

4. The method according to claim 3, characterized in that Before displaying the augmented reality image, the method further includes: Acquire scanning trajectory data matching the target object; A scanning trajectory line is drawn in the augmented reality image based on the scanning trajectory data, wherein the scanning trajectory line is used to guide the user to use the three-dimensional scanning device to perform a three-dimensional scan on the target object.

5. The method according to claim 3, characterized in that: The target object is an oral cavity, the virtual image is a three-dimensional model of an oral cavity reconstructed in real time, and the initial matching relationship is optimized based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship, including: Performing semantic segmentation processing on the three-dimensional model to obtain multiple tooth models; Performing semantic segmentation processing on the target object in the current frame real scene image to obtain multiple tooth regions; Determining a correspondence between the plurality of tooth models and the plurality of tooth regions based on the tooth numbers; The initial matching relationship is optimized based on the corresponding relationship and the matching relationship between the three-dimensional model and the reference real scene image to obtain a target matching relationship.

6. The method according to claim 5, characterized in that The initial matching relationship is optimized based on the corresponding relationship and the matching relationship between the three-dimensional model and the reference real scene image to obtain a target matching relationship, including: Determining a tooth model and a tooth region corresponding to a same tooth from among the plurality of tooth models and the plurality of tooth regions based on the corresponding relationship; Extracting a first boundary line between teeth and gums from the tooth model, and extracting a second boundary line between teeth and gums from the tooth region; Projecting the first dividing line into the reference real scene image to obtain a projection line, and constructing a reprojection error between the projection line and the second dividing line; The initial matching relationship is optimized based on the reprojection error and the matching relationship between the three-dimensional model and the reference real scene image to obtain a target matching relationship, wherein the target matching relationship minimizes the reprojection error.

7. The method according to claim 3, characterized in that Before displaying the augmented reality image, the method further includes: A specific type of region is identified from the augmented reality image, and identification processing is performed on the specific type of region.

8. The method according to claim 3, characterized in that The scanning mode of the three-dimensional scanning device includes a high-precision scanning mode and a low-precision scanning mode, and the method further includes: When it is determined based on the current frame real scene image that the area currently scanned by the three-dimensional scanning device is an area of ​​a specific type, the three-dimensional scanning device is notified to switch the scanning mode to a high-precision scanning mode.

9. An image processing device, characterized in that: The device comprises: An acquisition module is used to acquire a virtual image and obtain a real scene image of the current frame from a video stream captured by an AR device; A matching module, used to determine an initial matching relationship between the virtual image and the current frame real scene image, and optimize the initial matching relationship based on the matching relationship between the virtual image and the reference real scene image to obtain a target matching relationship; wherein the reference real scene image is one or more frames of images before the current frame of the real scene image in the video stream; The overlay module is used to overlay the virtual image onto the real scene image of the current frame based on the target matching relationship to obtain an augmented reality image, so as to display the augmented reality image in the AR device.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and computer instructions stored in the memory and executable by the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images