Three-dimensional scanning method, device, apparatus and storage medium

Abstract

This disclosure provides a 3D scanning method, apparatus, device, and storage medium. In scenarios where multiple users simultaneously scan a target area, on the one hand, to avoid users constantly switching their gaze between the 3D scanning device and the terminal device during the scanning process, augmented reality images marking the scanned areas can be displayed on the AR device worn by the user. On the other hand, to allow each user to simultaneously understand the current scanning status of other users, each user's AR device can display augmented reality images of the areas scanned by themselves, as well as those scanned by other users. Thus, users can understand the real-time scanning status of other users based on the content displayed on their AR devices, and adjust their own scanning strategies accordingly, avoiding duplicate or missed scans.

Description

Technical Field

[0001] This disclosure relates to the field of 3D scanning technology, and in particular to a 3D scanning method, apparatus, device and storage medium. Background Technology

[0002] With the development of 3D scanning technology, it has been widely applied in industrial fields. In industrial applications, it's often necessary to scan large target areas to reconstruct their 3D models. For example, a floor or building might need to be scanned to reconstruct its 3D model. Given the typically large size of these target areas and the numerous objects they may contain, scanning each location individually with a single 3D scanner would be extremely time-consuming. Therefore, multiple users typically collaborate, each using a separate 3D scanner, to speed up the process. Currently, in this collaborative scenario, each user only sees the results of their own scan and cannot track the progress of others. This leads to issues like excessive duplicate scanning of certain areas, wasting manpower, or missed areas due to lack of scanning. Summary of the Invention

[0003] This disclosure provides a three-dimensional scanning method, apparatus, device, and storage medium.

[0004] According to a first aspect of the present disclosure, a three-dimensional scanning method is provided, the method being applicable to a scenario where at least two users simultaneously scan a target area using a three-dimensional scanning device, each of the at least two users wearing an AR device, the method comprising:

[0005] For each AR device, a first augmented reality image and at least one frame of a second augmented reality image are acquired in real time for display on that AR device. The first augmented reality image is obtained by marking the scanned areas in the real-world image captured in real time by the AR device. Each frame of the second augmented reality image is obtained by marking the scanned areas in the real-world image captured in real time by one of the at least two AR devices worn by the user other than the AR device itself. The scanned areas in the real-world image captured by each AR device are determined by projecting a 3D model reconstructed in real time from data collected by a 3D scanning device used by the user wearing the AR device onto the real-world image.

[0006] The AR device displays the first augmented reality image and the at least one frame of the second augmented reality image.

[0007] According to a second aspect of the present disclosure, a three-dimensional scanning device is provided, the device being suitable for scenarios where at least two users simultaneously scan a target area using a three-dimensional scanning device, each of the at least two users wearing an AR device, the device comprising:

[0008] The acquisition module is configured to acquire, in real time, a first augmented reality image and at least one frame of a second augmented reality image for display on each AR device. The first augmented reality image is obtained by marking scanned areas in a real-world image captured in real time by the AR device. Each frame of the second augmented reality image is obtained by marking scanned areas in a real-world image captured in real time by one of the at least two AR devices worn by the user (excluding the AR device itself). The scanned areas in the real-world image captured by each AR device are determined by projecting a 3D model reconstructed in real time from data collected by a 3D scanning device used by the user wearing the AR device onto the real-world image.

[0009] A display module is used to display the first augmented reality image and the at least one frame of the second augmented reality image in the AR device.

[0010] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device including a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein when the processor executes the computer instructions, it can implement the method mentioned in the first aspect above.

[0011] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed, implement the method mentioned in the first aspect above.

[0012] The 3D scanning method provided in this disclosure, on the one hand, avoids the need for users to constantly switch their view between the 3D scanning device and the terminal device during the scanning process. It combines AR technology with 3D scanning technology, allowing each user to wear an AR device. The AR device's camera captures a real-world image of the user's current scanning area, and the data collected by the 3D scanning device is used to reconstruct a 3D model of the scanned area in real time. This 3D model is projected onto the real-world image, identifying the scanned area. The scanned area is then marked to obtain an augmented reality image, which is displayed on the AR device's screen. Thus, the user only needs to view the augmented reality image displayed on the AR device's screen to understand the current 3D model reconstruction status and adjust the position of the 3D scanning device to scan the target area. On the other hand, to allow each user to simultaneously understand the scanning status of other users, each user's AR device can display augmented reality images of areas scanned by other users, in addition to displaying their own scanned area. This allows users to understand the real-time scanning status of other users based on the content displayed on their AR devices, and adjust their own scanning strategy accordingly, avoiding excessive re-scanning or missed scans of the target area.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0015] Figure 1 This is a schematic diagram of a multi-person scanning scenario in existing technologies.

[0016] Figure 2 This is a schematic diagram of an application scenario according to an embodiment of this disclosure.

[0017] Figure 3 This is a flowchart of a three-dimensional scanning method according to an embodiment of the present disclosure.

[0018] Figure 4 This is a schematic diagram of the display interface of an AR device according to an embodiment of the present disclosure.

[0019] Figure 5 This is a schematic diagram of the display interface of an AR device in overall view mode and partial view mode according to an embodiment of the present disclosure.

[0020] Figure 6 This is a schematic diagram of an embodiment of the present disclosure of obtaining a stitched image by stitching together a first augmented reality image and a second augmented reality image.

[0021] Figure 7 This is a schematic diagram of the logical structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0022] Figure 8 This is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0024] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0025] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0026] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0027] With the development of 3D scanning technology, it has been widely applied in industrial fields. In industrial applications, it is often necessary to scan large target areas to reconstruct their 3D models. For example, a floor or building might need to be scanned to reconstruct its 3D model. Given that the target area is typically large and may contain many objects, scanning each location individually with a single 3D scanning device would be extremely time-consuming. Therefore, it is common practice for multiple users to work together, each using a separate 3D scanning device, to speed up the scanning process.

[0028] like Figure 1 As shown, in current scenarios where multiple users collaborate to scan a large area, each user typically carries a 3D scanning device and a terminal device (such as a mobile phone, tablet, or laptop). While scanning the target area, the 3D scanning device sends the collected data to the terminal device. The terminal device then performs real-time 3D reconstruction of the scanned area based on this data, creating a 3D model of the scanned area, which is displayed on the terminal device. Users can then understand the real-time status of their scanned area based on the model displayed on the terminal device. However, because each user needs to adjust the position of the 3D scanning device based on the reconstructed 3D model during the scanning process, they must simultaneously view the reconstructed 3D model on the terminal device and the 3D scanning device, requiring them to switch their gaze back and forth between the two, causing significant inconvenience during the scanning process.

[0029] Furthermore, each user can only see the real-time status of the area they scanned and does not know the scanning status of others. Therefore, it is impossible to know whether a certain area has been scanned by others. This may lead to multiple users scanning the same area repeatedly, resulting in duplicate work and wasted manpower. Alternatively, there may be areas that no one has scanned, leading to missed scans.

[0030] Based on this, the present disclosure provides a three-dimensional scanning method. On the one hand, to avoid users having to constantly switch their viewpoints between the three-dimensional scanning device and the terminal device during the scanning process, AR (Augmented Reality) technology can be combined with three-dimensional scanning technology. That is, each user can wear an AR device (e.g., AR glasses), and the AR device's camera can capture a real-world image of the user's current scanning area. The data collected by the three-dimensional scanning device is used to reconstruct a three-dimensional model of the scanned area in real time. The three-dimensional model is then projected onto the real-world image to determine the scanned area in the real-world image. The scanned area is then marked to obtain an augmented reality image, which is displayed on the AR device's screen. Thus, the user only needs to look at the augmented reality image displayed on the AR device's screen to understand the current three-dimensional model reconstruction status and adjust the position of the three-dimensional scanning device to scan the target area.

[0031] On the other hand, in order to allow each user to understand the current scanning status of other users at the same time, each user's AR device can display augmented reality images of the areas scanned by themselves as well as augmented reality images of the areas scanned by other users. Thus, users can understand the current real-time scanning status of other users based on the content displayed by the AR device, and then adjust their own scanning strategy to avoid duplicate scanning or missed scanning.

[0032] In some scenarios, the method of this disclosure embodiment can be executed by an AR device. For example, each user's AR device can acquire data collected by the 3D scanning device used by the user, reconstruct a 3D model using the data, and the AR device can also capture a real-world image of the user's currently scanned area using a camera. The 3D model can then be projected onto the real-world image to determine the scanned area in the real-world image, and the scanned area can be marked to obtain an augmented reality image of the AR device (hereinafter referred to as the first augmented reality image). Simultaneously, each user's AR device can communicate and connect with other users' AR devices and receive augmented reality images from other AR devices (hereinafter referred to as the second augmented reality image). The first and second augmented reality images are then displayed on the screen of the AR device.

[0033] In some scenarios, the method of this disclosure embodiment can also be executed by a designated device (e.g., a laptop, tablet, cloud server, etc.). This device can communicate and connect with each user's AR device and 3D scanning device to acquire real-world images collected by each user's AR device and data collected by each user's 3D scanning device. Then, it can use this data to generate augmented reality images corresponding to each user, and then send all augmented reality images (augmented reality images of this AR device and augmented reality images of other AR devices) to each user's AR device so that each user's AR device can display all the augmented reality images.

[0034] Alternatively, in some scenarios, some steps of the method can be performed by the AR device, and some steps can be performed by the designated device. The specific steps can be flexibly set based on actual needs, and this disclosure does not impose any restrictions.

[0035] Alternatively, in some scenarios, users can move 3D scanning devices not only by hand but also by mounting them on remote-controlled vehicles or drones.

[0036] like Figure 2The diagram illustrates an application scenario of this disclosure. For example, multiple users (two users in the diagram) can wear AR glasses and hold 3D scanning devices to collaboratively scan a floor in 3D to reconstruct a 3D model of that floor. Each user's AR glasses and 3D scanning device are connected to a cloud server. For instance, user 1's AR glasses can send its real-time captured real-scene image 1 to the cloud server, and user 1's 3D scanning device can also send its real-time captured scan data 1 to the cloud server. The cloud server can reconstruct a 3D model 1 based on the scan data 1, determine the matching relationship between the 3D model 1 and the real-scene image 1, project the 3D model 1 onto the real-scene image 1 based on the matching relationship, identify the scanned areas in the real-scene image 1, and mark the scanned areas to obtain the augmented reality image 1. Similarly, User 2's AR glasses can send its real-time captured real-scene image 2 to the cloud server, and User 2's 3D scanning device can also send its real-time captured scan data 2 to the cloud server. The cloud server can reconstruct a 3D model 2 based on the scan data 2, project the 3D model 2 onto the real-scene image 2, determine the scanned areas in the real-scene image 2, and mark the scanned areas to obtain augmented reality image 2. The cloud server can then send augmented reality image 1 and augmented reality image 2 to User 1's AR glasses and User 2's AR glasses respectively, so that both users' AR glasses can display the two frames of augmented reality images. Thus, each user can understand their own real-time scanning status and the real-time scanning status of other users through the content displayed on their AR glasses.

[0037] The AR device in this embodiment may include a camera and a display screen. The camera can be used to capture real-world images, and the display screen can be used to display generated augmented reality images. The AR device may be AR glasses or other AR products with similar functions.

[0038] The target area in this embodiment can be any type of area that needs to be reconstructed in three dimensions, such as a floor, building, pipeline, industrial product, industrial equipment, cultural relic, automobile, etc.

[0039] The method provided in this disclosure is applicable to scenarios where at least two users simultaneously scan a target area using a 3D scanning device, wherein each of the at least two users wears an AR device.

[0040] like Figure 3 As shown, the method may include the following steps:

[0041] S302. For each AR device, acquire in real time a first augmented reality image for display on the AR device and at least one frame of a second augmented reality image, wherein the first augmented reality image is obtained by marking the scanned area in the real-world image captured in real time by the AR device, and each frame of the second augmented reality image is obtained by marking the scanned area in the real-world image captured in real time by one of the at least two AR devices worn by the user other than the AR device itself; the scanned area in the real-world image captured by each AR device is determined by projecting a three-dimensional model reconstructed in real time from data collected by a three-dimensional scanning device used by the user wearing the AR device onto the real-world image;

[0042] In step S302, during the process of at least two users simultaneously scanning the target area using a 3D scanning device, for each user's AR device, a first augmented reality image and at least one frame of a second augmented reality image can be acquired in real time for display on that AR device. The first augmented reality image is used to display the real-time scanning status of the area where the user is currently located, such as which areas are scanned and which are unscanned. When generating the first augmented reality image, the real-world image of the user's currently scanned area collected by the AR device and the data collected by the 3D scanning device used by the user can be acquired in real time. This data is then used to perform real-time 3D reconstruction of the scanned area to obtain a 3D model. The matching relationship between the 3D model and the real-world image can then be determined. Based on this matching relationship, the 3D model is projected onto the real-world image, and the projection area of ​​the 3D model in the real-world image is determined, which is also the currently scanned area. The scanned area can then be marked to obtain the first augmented reality image.

[0043] Each frame of the second augmented reality image is used to display the real-time situation of the area currently scanned by one of the other users besides the current user. When generating the second augmented reality image, the real-world image of the area currently scanned by the other user's AR device can be acquired in real time, as well as data acquired in real-time by the other user's 3D scanning device. This data is then used to perform real-time 3D reconstruction of the scanned area to obtain a 3D model. This 3D model can then be projected onto the real-world image to determine and mark the scanned area, resulting in the second augmented reality image. One frame of the second augmented reality image can be generated for each other user using the same method.

[0044] S304. Display the first augmented reality image and the at least one frame of the second augmented reality image in the AR device.

[0045] In step S304, after acquiring the first augmented reality image and at least one frame of the second augmented reality image, the first augmented reality image and at least one frame of the second augmented reality image can be displayed in real time on the AR device. For example, if the method is executed by the AR device, the first augmented reality image and at least one frame of the second augmented reality image can be directly displayed on the AR device's screen. If the method is executed by another device (e.g., a cloud server) that is communicatively connected to the AR device, the other device can send the acquired first augmented reality image and at least one frame of the second augmented reality image to the AR device so that the images can be displayed on the AR device.

[0046] Of course, the same approach can be used for each user's AR device to display the scanning status of the area scanned by that user, as well as the scanning status of areas scanned by other users. This allows each user to easily understand the current scanning status of other users in real time, such as which areas other users have scanned, which areas have not been scanned, and the scanning quality of the scanned areas, etc.

[0047] Furthermore, when displaying a first augmented reality image and at least one frame of a second augmented reality image, the AR device can display these images through multiple display windows on the display interface, with each display window capable of displaying one frame of the image. In some scenarios, such as... Figure 4 As shown, considering the limited size of the display interface, a main display window and multiple preview windows can be set up on the display interface. The area occupied by the main display window can be larger than that occupied by the preview windows. When a user wants to clearly view a specific frame of image, they can move that image to the main display window for display. For example, during the scanning process, users spend most of their time focusing on the scanning status of their own scanned area. Therefore, the first augmented reality image can be displayed in the main display window, and each preview window can display one frame of the second augmented reality image. When a user wants to see the scanning status of other users' scanned areas, they can click on the second augmented reality image corresponding to that other user, switching the second augmented reality image to the main display window for display, so that they can clearly see the scanning status of other users' scanned areas.

[0048] Considering scenarios involving scanning large target areas, users typically need to continuously move around, scanning each sub-region within the target area one by one. That is, at each user's location, the real-world image captured by the AR device is only an image of the currently scanned area, i.e., a local image of the target area. The 3D model constructed from the data currently collected by the 3D scanning device is also a local 3D model corresponding to that local image. As the user gradually moves to scan different areas, the content of the real-world image captured by the AR device constantly changes, and there are overlapping areas between adjacent images, facilitating the stitching together of areas already scanned by the user. Similarly, the data collected by the 3D scanning device also has overlapping areas, facilitating the stitching together of 3D models of areas already scanned by the user.

[0049] Considering that during the scanning process, users may sometimes only want to view the scanned local area, and at other times they may want to view the scanned entire area. For example... Figure 5 As shown, to meet users' needs in different scenarios, two display modes can be set for AR devices: overall view mode and partial view mode. In partial view mode, the augmented reality image displayed by the AR device only reflects the scanning situation of the local area currently scanned by the user. That is, the real-world image used to generate the first augmented reality image and / or the second augmented reality image is the local real-world image currently captured by the AR device, and the scanned area in the first augmented reality image and / or the second augmented reality image is also the scanned area in this local real-world image.

[0050] In the overall view mode, the augmented reality image displayed by the AR device reflects the scanning status of all areas that the user has scanned. That is, the real-world image used to generate the first augmented reality image and / or the second augmented reality image is a panoramic image obtained by stitching together all the local real-world images that the AR device has captured, and the scanned areas in the first augmented reality image and / or the second augmented reality image are all the scanned areas in the panoramic image.

[0051] By setting up two display modes, users can easily view the scanning status of the current local scanning area through AR devices, and also view the overall scanning status of the currently scanned area through AR devices, making it more convenient and flexible.

[0052] In the process of multiple users collaborating to scan a target area, the users typically start from different locations within the target area and scan sequentially in a certain direction. For example... Figure 6As shown, suppose user A and user B start scanning from different sides of the target area and gradually move towards the other side. When user A or user B scans to a certain position, the real-world image captured by user A's AR device and the real-world image captured by user B's AR device will have overlapping areas. In this case, to better reflect the connection between the areas scanned by user A and user B, and to display the scanning situation of the target area from an overall perspective, the real-world images captured by the two AR devices can be stitched together before displaying. Therefore, in some embodiments, when it is determined that there is an overlapping area between the first augmented reality image and any frame of the second augmented reality image, the first augmented reality image and the second augmented reality image can be stitched together based on the overlapping area to obtain a stitched image, which is then displayed on the AR device. Through this stitched image, users can understand their own and others' scanning situations in one image, and can also associate their own scanning area with the scanning areas of others.

[0053] In some embodiments, if the 3D models constructed based on data collected by two users' 3D scanning devices have overlapping areas, it indicates that the scanned areas of the two users overlap. Since the local 3D models obtained by different users need to be stitched together to form a global 3D scanning model, a certain degree of overlap is necessary to facilitate the stitching of the local 3D models obtained by different users. However, if the overlapping area in the scanned areas is too large, it will not only fail to improve the accuracy of the stitching result but also lead to multiple users scanning the same area excessively, resulting in too much repetitive work and wasted resources.

[0054] Therefore, in some embodiments, if it is determined that there is an overlap between the scanned area in the first augmented reality image and the scanned area in any frame of the second augmented reality image, the overlapping area is marked so that the user knows that their current scanned area overlaps with the scanned area of ​​others, and the 3D model can be stitched together based on the overlapping area. At the same time, the size of the overlapping area can also be determined based on the marked image.

[0055] In some embodiments, considering that when stitching scanned regions in two augmented reality images, a larger overlapping area does not necessarily lead to more accurate stitching, i.e., a smaller stitching error, a preset area threshold can be set. If the area of ​​the overlapping region in the scanned region exceeds this preset area threshold, a prompt message is issued to the user so that the user can stop repeatedly scanning that region, avoiding redundant work and wasting resources. This preset area threshold can be set based on the stitching error. That is, when the area of ​​the overlapping region exceeds the preset area threshold, as the area of ​​the overlapping region increases, the reduction in stitching error when stitching the scanned regions in the first augmented reality image and the second augmented reality image becomes very slow, less than a preset threshold, or the stitching error does not continue to decrease. In this case, it indicates that the area of ​​the overlapping region is sufficient to achieve accurate stitching of the scanned regions, and the user does not need to repeat the scan. Therefore, a prompt message can be issued to the user to stop the repetitive work.

[0056] For example, when users are scanning a target area while constantly moving, and two users are close together, the real-world images captured by their AR devices often overlap. Based on this overlap, the real-world images captured by the users' AR devices can be stitched together to obtain a stitched image. As users continue scanning, the scanned areas of the 3D scanning devices may also overlap. For example, the 3D model constructed from the data collected by each user's 3D scanning device can be projected onto this stitched image to obtain the projection area of ​​the 3D model in the stitched image (i.e., the currently scanned area). Then, it can be determined whether the two projection areas overlap. If they overlap, it means that the scanned areas overlap, and the overlapping projection area can be identified. Then, it can be further determined whether the area of ​​the overlapping projection area is greater than a preset area threshold. If it is, a prompt is issued to the user so that the user can stop the overlapping scan.

[0057] In some embodiments, the 3D scanning device may include two scanning modes: a high-precision scanning mode and a low-precision scanning mode. In the high-precision scanning mode, the deviation between the 3D points measured by the 3D scanning device and the actual 3D points of the object is small, i.e., the measurement error is small. In the low-precision scanning mode, the deviation between the 3D points measured by the 3D scanning device and the actual 3D points of the object is large, i.e., the measurement error is large. Considering that certain areas of the target object may be areas that users need to focus on, these areas require more refined measurements and have higher error requirements. For example, when scanning a pipe, for crack areas in the pipe, in order to accurately determine the size of the crack in order to take appropriate treatment measures based on the crack size, these areas need to be scanned more refinedly. Therefore, after acquiring a real-world image, the AR device can determine whether the area currently being scanned by the 3D scanning device is a specific type of area based on the real-world image. For example, based on the matching relationship between the reconstructed 3D model and the real-world image, it can determine which area of ​​the real-world image the current 3D model corresponds to. This area is the area currently being scanned. Then, the trained model can be used to identify this area and determine whether it is a specific type of area. If so, the 3D scanning device can be notified to switch the scanning mode to a high-precision scanning mode to perform a more detailed scan of the area.

[0058] Typically, during the scanning process, users may focus on specific types of areas within the target region that they are particularly interested in. Therefore, these areas can be marked for easy viewing. For example, the target region might include pipes, and these specific types of areas could be areas with cracks or areas of rust and corrosion within the pipes. Thus, in some embodiments, before displaying the first augmented reality image, specific types of areas can be identified from the first augmented reality image and marked accordingly.

[0059] For scenarios involving large-scale scanning, the target area may typically include multiple scenes. Considering that there are often optimal scanning trajectories for different scene types, scanning the current scene based on these optimal trajectories can achieve both a more comprehensive scan and improved scanning efficiency. Therefore, in some embodiments, to facilitate user scanning, the first augmented reality image may include scanning trajectory lines to guide the user in scanning the target area. For example, for different scene types (such as floors, doors and windows, various furniture, etc.), optimal scanning trajectory lines can be pre-designed for each scene type, and the data related to the scanning trajectory lines, i.e., scanning trajectory line data, can be stored. This scanning trajectory line data is used to indicate the position and direction information of the scanning trajectory lines. After acquiring a real-world image of the current scanning area using an AR device, the scene type of the scanned area can be identified based on the real-world image. Then, scan trajectory data matching that scene type is obtained, and the scan trajectory line is drawn in the real-world image based on this data. The real-world image with the drawn scan trajectory line is then overlaid with a 3D model, resulting in a first augmented reality image that also includes the scan trajectory line. Users can move the 3D scanning device along the scan trajectory line in this first augmented reality image to scan the target area. This method significantly improves scanning efficiency, facilitates user operation, and allows for accurate scanning of scenes the user has not previously encountered, yielding better results.

[0060] In some scenarios, certain target areas may need to be scanned multiple times to determine their changing trends over time. For example, for certain pipelines, they can be scanned at regular intervals to determine changes in corrosion location or cracks over time. Therefore, in some embodiments, after reconstructing a 3D model of the target area in real time based on the current scan data, a historical 3D model reconstructed based on historical scan data of the target area can also be obtained. Then, the historical 3D model can be compared with the currently reconstructed 3D model to determine whether there are any risk areas, such as areas where pipeline cracks are rapidly increasing. These risk areas can then be marked in the real-world image so that they can be displayed to the user through AR devices, and the user can also be alerted.

[0061] It is easy to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space limitations, each solution in this disclosure will not be listed individually.

[0062] Accordingly, embodiments of this disclosure also provide a three-dimensional scanning device, such as... Figure 7As shown, the device is suitable for scenarios where at least two users simultaneously scan a target area using a 3D scanning device, each of the at least two users wearing an AR device, and the device 70 includes:

[0063] The acquisition module 71 is configured to acquire, in real time, a first augmented reality image and at least one frame of a second augmented reality image for display on each AR device. The first augmented reality image is obtained by marking the scanned areas in the real-world image captured in real time by the AR device. Each frame of the second augmented reality image is obtained by marking the scanned areas in the real-world image captured in real time by one of the at least two AR devices worn by the user (excluding the AR device itself). The scanned areas in the real-world image captured by each AR device are determined by projecting a 3D model reconstructed in real time from data collected by a 3D scanning device used by the user wearing the AR device onto the real-world image.

[0064] Display module 72 is used to display the first augmented reality image and the at least one frame of the second augmented reality image in the AR device.

[0065] The specific steps of the task processing method performed by the above-mentioned device can be referred to the description in the above method embodiments, and will not be repeated here.

[0066] Furthermore, embodiments of this disclosure also provide a device, such as... Figure 8 As shown, the device includes a processor 81, a memory 82, and computer instructions stored in the memory 82 that can be executed by the processor 81. When the processor 81 executes the computer instructions, it implements the method described in any of the above embodiments.

[0067] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0068] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0069] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0070] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0071] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0072] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A three-dimensional scanning method, characterized in that, The method is applicable to scenarios where at least two users simultaneously scan a target area using a 3D scanning device, each of the at least two users wearing an AR device. The method includes: For each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device are acquired in real time. The first augmented reality image is obtained by marking the scanned area in the real-world image captured in real time by the AR device. Each frame of the second augmented reality image is obtained by marking the scanned area in the real-world image captured in real time by one of the at least two AR devices worn by the user (excluding the AR device itself). The scanned area in the real-world image captured by each AR device is determined as follows: real-time 3D reconstruction is performed on the data collected by the 3D scanning device used by the user wearing the AR device; the matching relationship between the real-time reconstructed 3D model and the real-world image is determined; based on the matching relationship, the 3D model is projected onto the real-world image; and the projection area of ​​the 3D model in the real-world image is determined as the scanned area. The AR device displays the first augmented reality image and the at least one frame of the second augmented reality image. Each AR device's display mode includes an overall view mode and a partial view mode; wherein, in the overall view mode, the real-world image in the first augmented reality image and / or the second augmented reality image is a panoramic image obtained by stitching together all real-world images captured by the AR device, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the panoramic image. In the local view mode, the real-world image in the first augmented reality image and / or the second augmented reality image is a local image captured by the AR device at the current moment, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the local image.

2. The method according to claim 1, characterized in that, The method further includes: If it is determined that there is an overlapping area between the first augmented reality image and any frame of the second augmented reality image, the first augmented reality image and the frame of the second augmented reality image are stitched together based on the overlapping area to obtain a stitched image, and the stitched image is displayed in the AR device.

3. The method according to claim 1 or 2, characterized in that, The method further includes: When it is determined that there is an overlapping area between the scanned area in the first augmented reality image and the scanned area in any frame of the second augmented reality image, the overlapping area is marked.

4. The method according to claim 3, characterized in that, After marking the overlapping region, the method further includes: Determine whether the area of ​​the overlapping region reaches a preset area threshold; If so, a notification message will be sent to the user; Wherein, after the area of ​​the overlapping region reaches a preset area threshold, as the area of ​​the overlapping region increases, the decrease in stitching error when stitching the scanned area in the first augmented reality image and the scanned area in the second augmented reality image is less than a preset magnitude.

5. The method according to claim 1, characterized in that, The display mode of the AR device can be switched based on the mode switching command input by the user.

6. The method according to claim 1, characterized in that, The AR device's display interface includes a main display window and at least one preview window. The first augmented reality image is displayed in the main display window, and the area of ​​the main display window is larger than the area of ​​the preview window. Each frame of the second augmented reality image is displayed in one of the preview windows. The method further includes: In response to a user dragging a second augmented reality image from one of the preview windows to the main display window, the dragged second augmented reality image is displayed in the main display window, and the first augmented reality image is displayed in one of the preview windows.

7. The method according to claim 1, characterized in that, Each 3D scanning device includes a high-precision scanning mode and a low-precision scanning mode, and the method further includes: If, based on the real-world image captured by the AR device worn by the user, it is determined that the area being scanned by the 3D scanning device used by the user is a specific type of area, the 3D scanning device is notified to switch the scanning mode to high-precision scanning mode.

8. The method according to claim 1, characterized in that, The method further includes: Identify specific types of regions from the first augmented reality image and perform labeling processing on the specific types of regions.

9. The method according to claim 1, characterized in that, The first augmented reality image contains scanning trajectory lines to guide the user in scanning the target area, wherein the scanning trajectory lines are drawn in the following manner: Identify the scene type of the currently scanned area from the real-world images captured by the AR device. Retrieve scan trajectory data that matches the scene type from pre-stored scan trajectory lines; The scan trajectory line is drawn in the real-world image based on the scan trajectory data.

10. The method according to claim 1, characterized in that, The method further includes: A historical 3D model of the target area is obtained, which is reconstructed based on historical scan data of the target area. The historical 3D model is compared with the real-time reconstructed 3D model to determine the risk areas in the target area; The risk area is identified in the first augmented reality image.

11. A three-dimensional scanning device, characterized in that, The device is suitable for scenarios where at least two users simultaneously scan a target area using a 3D scanning device, each of the at least two users wearing an AR device, and the device includes: The acquisition module is configured to acquire, in real time, a first augmented reality image and at least one frame of a second augmented reality image for display on each AR device. The first augmented reality image is obtained by marking scanned areas in real-time images captured by the AR device. Each frame of the second augmented reality image is obtained by marking scanned areas in real-time images captured by one of the at least two AR devices worn by the user (excluding the AR device itself). The scanned areas in the real-time images captured by each AR device are determined as follows: real-time 3D reconstruction is performed on data collected by a 3D scanning device used by the user wearing the AR device; the matching relationship between the reconstructed 3D model and the real-time image is determined; based on the matching relationship, the 3D model is projected onto the real-time image; and the projection area of ​​the 3D model in the real-time image is determined as the scanned area. A display module is used to display the first augmented reality image and the at least one frame of the second augmented reality image through the AR device; Each AR device's display mode includes an overall view mode and a partial view mode; wherein, in the overall view mode, the real-world image in the first augmented reality image and / or the second augmented reality image is a panoramic image obtained by stitching together all real-world images captured by the AR device, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the panoramic image. In the local view mode, the real-world image in the first augmented reality image and / or the second augmented reality image is a local image captured by the AR device at the current moment, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the local image.

12. An electronic device, characterized in that, The electronic device includes a processor, a memory, and computer instructions stored in the memory that are executable by the processor. When the processor executes the computer instructions, it implements the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-10.

Images