Three-dimensional scanning method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional scanning method and device, equipment and a storage medium. In a scene in which a plurality of users scan a target area at the same time, on one hand, in order to avoid the situation that the users need to continuously switch sight lines back and forth between a three-dimensional scanning device and a terminal device in the scanning process, an augmented reality image marked with a scanned area can be displayed in an AR device worn by the users. And on the other hand, in order to enable each user to know the current scanning conditions of other users at the same time, in the AR equipment worn by each user, the augmented reality image of the area scanned by the user can be displayed, and the augmented reality images of the areas scanned by other users can also be displayed, so that the user experience is improved. Therefore, the user can know the current real-time scanning condition of other users based on the content displayed by the AR equipment, so that the scanning strategy of the user is adjusted, and repeated scanning or missing scanning is avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional scanning technology, and in particular to a three-dimensional scanning method, device, equipment and storage medium. Background Art

[0002] With the development of three-dimensional scanning technology, three-dimensional scanning technology has been widely used in the industrial field. Considering that in the industrial field, it is usually necessary to scan a large range of target areas to reconstruct the three-dimensional model of the target area. For example, it may be necessary to perform a three-dimensional scan on a certain floor or a certain building to reconstruct the three-dimensional model of the floor or the building. Considering that the range of the target area is usually large, the target area may include more objects. If only one user uses a three-dimensional scanning device to scan each position of the target area one by one, it will take a long time. Therefore, it is usually a case of multiple users each using a three-dimensional scanning device to scan the target area together to speed up the scanning speed. At present, in this scenario where multiple people cooperate to scan a large area, each user can only understand the scanning status of the area scanned by himself, and does not know the scanning status of others, so it is impossible to know whether each area has been scanned by others. As a result, there will be too many repeated scans of certain areas, wasting manpower, or there will be a problem of missing scans if no one scans a certain area. Summary of the invention

[0003] The present invention provides a three-dimensional scanning method, device, equipment and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a three-dimensional scanning method is provided, the method being applicable to a scenario in which at least two users simultaneously scan a target area using a three-dimensional scanning device, each of the at least two users wearing an AR device, the method comprising:

[0005] For each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device are acquired in real time, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time from data collected by a three-dimensional scanning device used by a user wearing the AR device onto the real-scene image;

[0006] The first augmented reality image and the at least one frame of the second augmented reality image are displayed in the AR device.

[0007] According to a second aspect of an embodiment of the present disclosure, a three-dimensional scanning device is provided, the device being applicable to a scenario in which at least two users simultaneously use a three-dimensional scanning device to scan a target area, each of the at least two users wearing an AR device, the device comprising:

[0008] An acquisition module is used to acquire, for each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device in real time, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time from data collected by a three-dimensional scanning device used by the user wearing the AR device onto the real-scene image;

[0009] The display module is used to display the first augmented reality image and the at least one frame of the second augmented reality image through the AR device.

[0010] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein when the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0011] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0012] The three-dimensional scanning method provided by the embodiment of the present disclosure, on the one hand, in order to avoid the user from constantly switching the line of sight between the three-dimensional scanning device and the terminal device during the scanning process, the AR technology can be combined with the three-dimensional scanning technology, that is, each user can wear an AR device, collect the real-life image of the user's current scanning area through the camera of the AR device, and use the data collected by the three-dimensional scanning device to reconstruct the three-dimensional model of the scanned area in real time, project the three-dimensional model into the real-life image, determine the scanned area in the real-life image, mark the scanned area, obtain an augmented reality image, and display it on the display screen of the AR device, so that the user only needs to look at the augmented reality image displayed on the display screen of the AR device to understand the current three-dimensional model reconstruction situation, and adjust the position of the three-dimensional scanning device to scan the target area. On the other hand, in order to allow each user to understand the current scanning situation of other users at the same time, in the AR device worn by each user, in addition to displaying the augmented reality image of the area scanned by the user, the augmented reality image of the area scanned by other users can also be displayed, so that the user can understand the current real-time scanning situation of other users based on the content displayed by the AR device, and then adjust his own scanning strategy to avoid excessive repeated scanning or missed scanning of the target scanning area.

[0013] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0015] Figure 1 It is a schematic diagram of a multi-person scanning scene in the prior art.

[0016] Figure 2 It is a schematic diagram of an application scenario of an embodiment of the present disclosure.

[0017] Figure 3 It is a flow chart of a three-dimensional scanning method according to an embodiment of the present disclosure.

[0018] Figure 4 It is a schematic diagram of a display interface of an AR device according to an embodiment of the present disclosure.

[0019] Figure 5 It is a schematic diagram of a display interface of an AR device in an overall viewing mode and a local viewing mode according to an embodiment of the present disclosure.

[0020] Figure 6 It is a schematic diagram of an embodiment of the present disclosure, in which a first augmented reality image and a second augmented reality image are spliced ​​to obtain a spliced ​​image.

[0021] Figure 7 It is a schematic diagram of the logical structure of a three-dimensional scanning device according to an embodiment of the present disclosure.

[0022] Figure 8 It is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0024] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0025] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0027] With the development of 3D scanning technology, 3D scanning technology has been widely used in the industrial field. Considering that in the industrial field, it is usually necessary to scan a large target area to reconstruct the 3D model of the target area. For example, it may be necessary to perform a 3D scan on a certain floor or a certain building to reconstruct the 3D model of the floor or the building. Considering that the scope of the target area is usually large and the target area may include many objects, it takes a long time if only one user uses a 3D scanning device to scan each position of the target area one by one. Therefore, it is usually necessary for multiple users to use a 3D scanning device to scan the target area together to speed up the scanning speed.

[0028] like Figure 1 As shown, currently in the scenario where multiple people collaborate to scan a large area, each user usually holds a 3D scanning device and a terminal device (for example, a mobile phone, a tablet, or a laptop). When the user uses the 3D scanning device to scan the target area, the 3D scanning device can send the collected data to the terminal device, and the terminal device can perform real-time 3D reconstruction of the scanned area based on the collected data to obtain a 3D model of the scanned area and display it on the terminal device, so that the user can know the real-time situation of the area scanned by the user based on the model displayed by the terminal device. In addition, since each user needs to adjust the position of the 3D scanning device based on the reconstructed 3D model during the scanning process, it is necessary to view the reconstructed 3D model on the terminal device while viewing the 3D scanning device, and the line of sight needs to switch back and forth between the two, which brings great inconvenience to the user during the scanning process.

[0029] In addition, each user can only understand the real-time situation of the area scanned by himself, and does not know the scanning situation of others, so he cannot know whether a certain area has been scanned by others. As a result, multiple users may repeatedly scan certain areas, resulting in repeated work and waste of manpower. Or there may be a problem that no one scans a certain area, resulting in the problem of missing scanning of certain areas.

[0030] Based on this, the embodiments of the present disclosure provide a three-dimensional scanning method. On the one hand, in order to avoid the user from constantly switching his / her line of sight between the three-dimensional scanning device and the terminal device during the scanning process, AR (Augmented Reality) technology can be combined with the three-dimensional scanning technology, that is, each user can wear an AR device (for example, AR glasses), and use the camera of the AR device to collect the real-life image of the user's current scanning area, and use the data collected by the three-dimensional scanning device to reconstruct the three-dimensional model of the scanned area in real time, and project the three-dimensional model into the real-life image to determine the scanned area in the real-life image, and mark the scanned area to obtain an augmented reality image, and display it on the display screen of the AR device, so that the user only needs to look at the augmented reality image displayed on the display screen of the AR device to understand the current three-dimensional model reconstruction status, and adjust the position of the three-dimensional scanning device to scan the target area.

[0031] On the other hand, in order to allow each user to understand the current scanning status of other users at the same time, the AR device worn by each user can not only display the augmented reality image of the area scanned by the user, but also the augmented reality image of the area scanned by other users. Therefore, the user can understand the current real-time scanning status of other users based on the content displayed by the AR device, and then adjust his own scanning strategy to avoid repeated scanning or missed scanning.

[0032] In some scenarios, the method of the disclosed embodiment can be executed by an AR device. For example, each user's AR device can obtain data collected by the three-dimensional scanning device used by the user, and use the data to reconstruct a three-dimensional model. The AR device can collect the real-life image of the user's current scanning area through a camera, and project the three-dimensional model into the real-life image to determine the scanned area in the real-life image, and mark the scanned area to obtain an augmented reality image of the AR device (hereinafter referred to as the first augmented reality image). At the same time, each user's AR device can communicate with other users' AR devices and receive augmented reality images of other AR devices (hereinafter referred to as the second augmented reality image), and then display the first augmented reality image and the second augmented reality image on the display screen of the AR device.

[0033] In some scenarios, the method of the disclosed embodiments may also be executed by a designated device (e.g., a laptop computer, a tablet computer, a cloud server, etc.), which may be communicatively connected to the AR device and the three-dimensional scanning device of each user, thereby obtaining the real-scene images captured by each user's AR device and the data captured by each user's three-dimensional scanning device, and then using these data to generate an augmented reality image corresponding to each user, and then sending all augmented reality images (augmented reality images of this AR device and augmented reality images of other AR devices) to each user's AR device, so that each user's AR device can display all augmented reality images.

[0034] Alternatively, in some scenarios, some steps of the method may be executed by an AR device, and some steps may be executed by the designated device. The specific configuration may be flexibly set based on actual needs, and the embodiments of the present disclosure do not limit this.

[0035] Alternatively, in some scenarios, users can use a 3D scanning device to scan by not only moving the 3D scanning device by hand, but also by installing the 3D scanning device on a remote-controlled vehicle or drone and then moving the 3D scanning device by controlling the remote-controlled vehicle or drone.

[0036] like Figure 2As shown, it is a schematic diagram of an application scenario of an embodiment of the present disclosure. For example, multiple users (two users are used as an example in the figure) can wear AR glasses and hold three-dimensional scanning devices, and then work together to perform three-dimensional scanning on a certain floor to reconstruct the three-dimensional model of the floor. Among them, each user's AR glasses and three-dimensional scanning device are both connected to the cloud server in communication. For example, the AR glasses of user 1 can send the real-life image 1 collected by itself in real time to the cloud server, and the three-dimensional scanning device of user 1 can also send the scanning data 1 collected by itself in real time to the cloud server. The cloud server can reconstruct the three-dimensional model 1 based on the scanning data 1, and determine the matching relationship between the three-dimensional model 1 and the real-life image 1. Based on the matching relationship, the three-dimensional model 1 is projected into the real-life image 1, the scanned area in the real-life image 1 is determined, and the scanned area is marked to obtain an augmented reality image 1. Similarly, the AR glasses of user 2 can send the real-scene image 2 collected by itself in real time to the cloud server, and the three-dimensional scanning device of user 2 can also send the scanning data 2 collected by itself in real time to the cloud server. The cloud server can reconstruct the three-dimensional model 2 based on the scanning data 2, and project the three-dimensional model 2 into the real-scene image 2, determine the scanned area in the real-scene image 2, and mark the scanned area to obtain the augmented reality image 2. The cloud server can send the augmented reality image 1 and the augmented reality image 2 to the AR glasses of user 1 and the AR glasses of user 2 respectively, so that the above two frames of augmented reality images can be displayed in the AR glasses of both users. Therefore, each user can understand both his own real-time scanning status and the real-time scanning status of other users through the content displayed by the AR glasses.

[0037] The AR device in the embodiment of the present disclosure may include a camera and a display screen. The camera can be used to capture real-scene images, and the display screen can be used to display generated augmented reality images. The AR device may be AR glasses, or other AR products with similar functions.

[0038] The target area of ​​the embodiment of the present disclosure may be various types of areas that need to be three-dimensionally reconstructed, for example, it may be a floor, a building, a pipeline, an industrial product, industrial equipment, a cultural relic, a car, etc.

[0039] The method provided by the embodiment of the present disclosure is applicable to a scenario in which at least two users simultaneously scan a target area using a three-dimensional scanning device, wherein each of the at least two users wears an AR device.

[0040] like Figure 3 As shown, the method may include the following steps:

[0041] S302, for each AR device, obtaining in real time a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time using data collected by a three-dimensional scanning device used by the user wearing the AR device onto the real-scene image;

[0042] In step S302, when at least two users simultaneously use a three-dimensional scanning device to scan the target area, for each user's AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device can be obtained in real time. The first augmented reality image is used to display the real-time scanning status of the area where the user is currently located, for example, which areas are scanned areas and which areas are unscanned areas. When generating the first augmented reality image, the real-life image of the user's current scanning area collected by the AR device can be obtained in real time, and the data collected by the three-dimensional scanning device used by the user can be obtained in real time, and the scanned area can be reconstructed in real time using these data to obtain a three-dimensional model, and then the matching relationship between the three-dimensional model and the real-life image can be determined, and the three-dimensional model can be projected into the real-life image based on the matching relationship, and the projection area of ​​the three-dimensional model in the real-life image is determined, which is also the current scanned area, and then the scanned area can be marked to obtain the first augmented reality image.

[0043] Each frame of the second augmented reality image is used to display the real-time situation of the area currently scanned by one of the other users except the user. When generating the second augmented reality image, the real-life image of the area currently scanned by the other user collected by the AR device of the other user can be obtained in real time, and the data collected by the three-dimensional scanning device used by the other user can be obtained in real time, and the scanned area can be reconstructed in real time using these data to obtain a three-dimensional model. The three-dimensional model can then be projected into the real-life image, the scanned area can be determined and marked, and the second augmented reality image can be obtained. For each other user, a frame of the second augmented reality image can be generated in the same way.

[0044] S304: Display the first augmented reality image and the at least one frame of the second augmented reality image through the AR device.

[0045] In step S304, after acquiring the first augmented reality image and at least one frame of the second augmented reality image, the first augmented reality image and at least one frame of the second augmented reality image can be displayed in real time in the AR device. For example, if the method is executed by the AR device, the first augmented reality image and at least one frame of the second augmented reality image can be directly displayed through the display screen of the AR device. If the method is executed by other devices (such as a cloud server) that are communicatively connected to the AR device, the other devices can send the acquired first augmented reality image and at least one frame of the second augmented reality image to the AR device so that the above images can be displayed on the AR device.

[0046] Of course, the same method can be used for each user's AR device to display the scanning status of the area scanned by the user and the scanning status of the areas scanned by other users in each user's AR device. This makes it convenient for each user to understand the current scanning status of other users in real time, such as which areas are currently scanned by other users, which areas are not scanned, the scanning quality of the scanned areas, etc.

[0047] In addition, when displaying the first augmented reality image and at least one frame of the second augmented reality image, the AR device can display the above images through multiple display windows of the display interface, and each display window can be used to display one frame of the image. Figure 4 As shown, considering the limited size of the display interface, a main display window and multiple preview windows can be set on the display interface, wherein the area occupied by the main display window can be larger than the area occupied by the preview window. When the user wants to clearly view a frame of an image, the image can be moved to the main display window for display. For example, during the scanning process, the user spends most of the time focusing on the scanning status of the area scanned by himself. Therefore, the first augmented reality image can be displayed in the main display window, and each preview window can display a frame of the second augmented reality image. When the user wants to know the scanning status of the area scanned by other users, he can click on the second augmented reality image corresponding to the other user, and switch the second augmented reality image to the main display window for display, so that the scanning status of the area scanned by other users can be clearly seen.

[0048] Considering the scenario of scanning a large target area, users usually need to keep moving to scan each sub-area in the target area one by one. That is, every time the user is in a position, the real-life image collected by the AR device is only the image of the currently scanned area, that is, the local image of the target area, and the three-dimensional model constructed by the data currently collected by the three-dimensional scanning device is also a local three-dimensional model corresponding to the local image. In the process of the user gradually moving his position to scan different areas, the content of the real-life image collected by the AR device is also constantly changing, and there are overlapping areas between adjacent images, which is convenient for splicing the areas that the user has scanned. Similarly, there are overlapping areas in the data collected by the three-dimensional scanning device, which is convenient for splicing the three-dimensional models of the areas scanned by the user.

[0049] Considering that during the scanning process, the user may sometimes only want to view the scanning status of the current local area, and sometimes may want to view the scanning status of all the currently scanned areas. Figure 5 As shown, in order to meet the needs of users in different scenarios, two display modes can be set for the AR device, the overall perspective mode and the local perspective mode. In the local perspective mode, the augmented reality image displayed by the AR device only reflects the scanning status of the local area currently scanned by the user, that is, the real scene image used to generate the above-mentioned first augmented reality image and / or the second augmented reality image is the local real scene image currently collected by the AR device, and the scanned area in the above-mentioned first augmented reality image and / or the second augmented reality image is also the scanned area in the local real scene image.

[0050] In the overall perspective mode, the augmented reality image displayed by the AR device reflects the scanning status of all areas scanned by the user, that is, the real scene image used to generate the above-mentioned first augmented reality image and / or the second augmented reality image is a panoramic image obtained by stitching together all local real scene images collected by the AR device, and the scanned areas in the above-mentioned first augmented reality image and / or the second augmented reality image are all scanned areas in the panoramic image.

[0051] By setting two display modes, it is convenient for users to clearly view the scanning status of the current local scanning area through the AR device, and it is also convenient for users to view the overall scanning status of the scanned area at the current moment through the AR device, which is more convenient and flexible.

[0052] When multiple users cooperate to scan a target area, they usually start from different positions in the target area and scan one by one in a certain direction. Figure 6As shown, assuming that user A and user B start from different sides of the target area and gradually scan to the other side, when user A or user B scans to a certain position, there will be an overlapping area between the real scene image captured by the AR device of user A and the real scene image captured by the AR device of user B. In this case, in order to better reflect the connection between the area currently scanned by user A and the area scanned by user B, and to display the scanning situation of the target area from an overall perspective, the real scene images captured by the two AR devices can be spliced ​​and then displayed. Therefore, in some embodiments, when it is determined that there is an overlapping area between the first augmented reality image and any frame of the second augmented reality image, the first augmented reality image and the second augmented reality image can be spliced ​​based on the overlapping area to obtain a spliced ​​image, and the spliced ​​image can be displayed in the AR device. Through the spliced ​​image, the user can understand the scanning situation of himself and others in one image, and can also associate the user's own scanning area with the scanning area of ​​others.

[0053] In some embodiments, if there is an overlapping area in the three-dimensional model constructed based on the data collected by the three-dimensional scanning devices of two users, it means that the scanned areas of the two users overlap. Since the local three-dimensional models scanned by different users need to be spliced ​​together to form a global three-dimensional scanning model. Therefore, there needs to be a certain overlapping area in the scanned areas of different users to facilitate the splicing of the local three-dimensional models scanned by different users. However, if the overlapping area in the scanned area is too large, it will not be able to improve the accuracy of the splicing result, and it will also cause multiple users to scan the same area too much, resulting in too much duplication of work and waste of resources.

[0054] Therefore, in some embodiments, if it is determined that the scanned area in the first augmented reality image overlaps with the scanned area in any frame of the second augmented reality image, the overlapping area is marked so that the user knows that the current scanned area of ​​the user and the scanned area of ​​others overlap, and the three-dimensional model can be spliced ​​based on the overlapping area. At the same time, the size of the overlapping area can also be determined based on the marked image.

[0055] In some embodiments, considering that when the scanned areas in two frames of augmented reality images are stitched together, it is not the case that the larger the overlapping area, the more accurate the stitching effect, that is, the smaller the stitching error. Therefore, an area threshold can also be set in advance. If the area of ​​the overlapping area in the scanned area is greater than the preset area threshold, a prompt message is sent to the user so that the user stops continuing to scan the area repeatedly to avoid repeated operations and waste resources. Among them, the preset area threshold can be set based on the stitching error, that is, when the area of ​​the overlapping area is greater than the preset area threshold, as the area of ​​the overlapping area increases, when the scanned area in the first augmented reality image and the scanned area in the second augmented reality image are stitched together, the stitching error decreases very slowly, less than the preset amplitude, or the stitching error will not continue to decrease. At this time, it means that the area of ​​the overlapping area is sufficient to achieve accurate stitching of the scanned area, and the user does not need to scan again. Therefore, a prompt message can be sent to the user to prompt the user to stop repeating the operation.

[0056] For example, when the user is constantly moving to scan the target area, when the distance between two users is close, the real-life images captured by the AR devices of the two users usually overlap. Based on the overlapping area, the real-life images captured by the user's AR device can be spliced ​​together to obtain a spliced ​​image. As users continue to scan, there may also be overlapping areas in the scanned areas of the three-dimensional scanning device. For example, a three-dimensional model constructed from data collected by the three-dimensional scanning devices of each user can be projected into the spliced ​​image to obtain the projection area of ​​the three-dimensional model in the spliced ​​image (that is, the area currently scanned by the three-dimensional scan), and then it can be determined whether the two projection areas overlap. If they overlap, it means that the scanning areas overlap. At this time, the overlapping projection area can be identified. Then it can be further determined whether the area of ​​the overlapping projection area is greater than the preset area threshold. If it is greater, a prompt is issued to the user so that the user can stop the overlapping scan.

[0057] In some embodiments, the three-dimensional scanning device may include two scanning modes, namely, a high-precision scanning mode and a low-precision scanning mode. In the high-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is small, that is, the measurement error is small, while in the low-precision scanning mode, the deviation between the three-dimensional points measured by the three-dimensional scanning device and the actual three-dimensional points of the object is large, that is, the measurement error is large. Considering that certain areas of the target object may be areas that the user needs to focus on, these areas require more refined measurements and have higher requirements for errors. For example, when scanning a pipeline, for crack areas in the pipeline, since the size of the cracks must be accurately determined in order to take corresponding treatment measures based on the size of the cracks, these areas need to be scanned more finely. Therefore, after the AR device acquires the real-life image, it can determine whether the area currently scanned by the 3D scanning device is a specific type of area based on the real-life image. For example, based on the matching relationship between the reconstructed 3D model and the real-life image, it can determine which area of ​​the real-life image the current 3D model corresponds to. This area is the area currently being scanned. The trained model can then be used to identify the area to determine whether it is a specific type of area. If so, the 3D scanning device can be notified to switch the scanning mode to a high-precision scanning mode to enable a more refined scan of the area.

[0058] Generally speaking, during the scanning process, some specific types of areas in the target area may be areas of concern to the user, so these areas can be marked for easy viewing by the user. For example, the target area may include a pipeline, and the specific type of area may be an area with cracks in the pipeline, or an area with rust and corrosion in the pipeline. Therefore, in some embodiments, before displaying the first augmented reality image, a specific type of area can be identified from the first augmented reality image, and the specific type of area can be marked.

[0059] For scenes with large-scale scanning, the target area may usually include multiple scenes. Considering that there are often better scanning trajectories for different types of scenes during the scanning process, scanning the current scene based on the better scanning trajectory can not only perform a more comprehensive scan, but also improve the scanning efficiency. Therefore, in some embodiments, in order to facilitate user scanning, the first augmented reality image may include a scanning trajectory line to guide the user to scan the target area. For example, for different scene types (such as the ground, doors and windows, various furniture, etc.), the best scanning trajectory line for this type of scene can be pre-designed, and data related to the scanning trajectory line, that is, the scanning trajectory line data, is stored. The scanning trajectory line data is used to indicate the position information and direction information of the scanning trajectory line. After using the AR device to collect the real-life image of the current scanning area, the scene type of the current scanning area can be identified based on the real-life image, and then the scanning track data matching the scene type can be obtained, and the scanning track line can be drawn in the real-life image based on the scanning track data, and then the real-life image with the scanning track line drawn can be superimposed with the three-dimensional model, so that the first augmented reality image finally generated also includes the scanning track line, and the user can move the three-dimensional scanning device along the scanning track line in the first augmented reality image to scan the target area. In this way, the scanning efficiency can be greatly improved, the user operation is convenient, and the user can accurately operate the scene that has not been scanned before, and obtain better scanning results.

[0060] In some scenarios, for certain target areas, it may be necessary to scan them multiple times to determine their changing trends over time. For example, for certain pipelines, they can be scanned at regular intervals to determine the changes in their corrosion positions over time, or the changes in cracks over time, etc. Therefore, in some embodiments, after the three-dimensional model of the target area is reconstructed in real time based on the current scanned data, a historical three-dimensional model reconstructed based on the historical scanned data of the target area can also be obtained, and then the historical three-dimensional model can be compared with the currently reconstructed three-dimensional model to determine whether there are some risk areas, such as areas where pipeline cracks increase sharply, and then these risk areas can be marked in the real-life image so that they can be displayed to the user through the AR device, and at the same time, the user can also be warned.

[0061] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0062] Accordingly, the present disclosure also provides a three-dimensional scanning device, such as Figure 7As shown, the device is suitable for a scenario where at least two users simultaneously use a three-dimensional scanning device to scan a target area, and each of the at least two users wears an AR device. The device 70 includes:

[0063] The acquisition module 71 is used to acquire, for each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device in real time, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time using data collected by a three-dimensional scanning device used by the user wearing the AR device onto the real-scene image;

[0064] The display module 72 is used to display the first augmented reality image and the at least one frame of the second augmented reality image through the AR device.

[0065] Among them, the specific steps of the above-mentioned device executing the task processing method can refer to the description in the above-mentioned method embodiment, and will not be repeated here.

[0066] Furthermore, the present disclosure also provides a device, such as Figure 8 As shown, the device includes a processor 81, a memory 82, and computer instructions stored in the memory 82 for execution by the processor 81. When the processor 81 executes the computer instructions, the method described in any one of the above embodiments is implemented.

[0067] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0068] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0069] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.

[0070] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0071] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0072] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.

Claims

1. A three-dimensional scanning method, characterized in that: The method is applicable to a scenario where at least two users simultaneously scan a target area using a three-dimensional scanning device, and each of the at least two users wears an AR device. The method includes: For each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device are acquired in real time, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time from data collected by a three-dimensional scanning device used by a user wearing the AR device onto the real-scene image; The first augmented reality image and the at least one frame of the second augmented reality image are displayed through the AR device.

2. The method according to claim 1, characterized in that The method further comprises: When it is determined that there is an overlapping area between the first augmented reality image and any frame of the second augmented reality image, the first augmented reality image and the second augmented reality image of the frame are spliced ​​based on the overlapping area to obtain a spliced ​​image, and the spliced ​​image is displayed in the AR device.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When it is determined that there is an overlapping area between the scanned area in the first augmented reality image and the scanned area in any frame of the second augmented reality image, the overlapping area is marked.

4. The method according to claim 3, characterized in that After marking the overlapping area, the method further includes: Determine whether the area of ​​the overlapping region reaches a preset area threshold; If yes, a prompt message is sent to the user; Among them, after the area of ​​the overlapping area reaches a preset area threshold, as the area of ​​the overlapping area increases, the decrease in the stitching error when stitching the scanned area in the first augmented reality image and the scanned area in the second augmented reality image is less than the preset amplitude.

5. The method according to claim 1, characterized in that The display mode of each AR device includes an overall viewing mode and a local viewing mode, and the display mode of the AR device can be switched based on a mode switching instruction input by a user; Wherein, in the overall viewing angle mode, the real scene image in the first augmented reality image and / or the second augmented reality image is a panoramic image obtained by stitching all real scene images collected by the AR device, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the panoramic image; In the local perspective mode, the real scene image in the first augmented reality image and / or the second augmented reality image is a local image captured by the AR device at the current moment, and the scanned area in the first augmented reality image and / or the second augmented reality image is the scanned area in the local image.

6. The method according to claim 1, characterized in that The display interface of the AR device includes a main display window and at least one preview window, the first augmented reality image is displayed in the main display window, the area of ​​the main display window is larger than the area of ​​the preview window, and each frame of the second augmented reality image is displayed in one of the preview windows, and the method further includes: In response to a user dragging a second augmented reality image in one of the preview windows to the main display window, the second augmented reality image dragged by the user is displayed in the main display window, and the first augmented reality image is displayed in one of the preview windows.

7. The method according to claim 1, characterized in that The scanning mode of each three-dimensional scanning device includes a high-precision scanning mode and a low-precision scanning mode, and the method further includes: When it is determined that the area currently scanned by the three-dimensional scanning device used by the user is a specific type of area based on the real scene image captured by the AR device worn by the user, the three-dimensional scanning device is notified to switch the scanning mode to a high-precision scanning mode.

8. The method according to claim 1, characterized in that The method further comprises: A specific type of area is identified from the first augmented reality image, and identification processing is performed on the specific type of area.

9. The method according to claim 1, characterized in that: A scanning trajectory line is drawn in the first augmented reality image for guiding the user to scan the target area, wherein the scanning trajectory line is drawn based on the following method: Identify the scene type of the current scanning area from the real-scene image collected by the AR device. Acquire scanning trajectory data matching the scene type from pre-stored scanning trajectory lines; The scanning trajectory line is drawn in the real scene image based on the scanning trajectory data.

10. The method according to claim 1, characterized in that The method further comprises: Acquire a historical three-dimensional model of the target area, wherein the historical three-dimensional model is reconstructed based on historical scanning data of the target area; Comparing the historical three-dimensional model with the three-dimensional model reconstructed in real time to determine the risk area in the target area; The risk area is identified in the first augmented reality image.

11. A three-dimensional scanning device, characterized in that: The device is suitable for a scenario where at least two users simultaneously scan a target area using a three-dimensional scanning device, and each of the at least two users wears an AR device. The device includes: An acquisition module is used to acquire, for each AR device, a first augmented reality image and at least one frame of a second augmented reality image for display on the AR device in real time, wherein the first augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by the AR device, and each frame of the second augmented reality image is obtained by marking a scanned area in a real-scene image collected in real time by one of the AR devices other than the AR device among the AR devices worn by the at least two users; the scanned area in the real-scene image collected by each AR device is determined by projecting a three-dimensional model reconstructed in real time from data collected by a three-dimensional scanning device used by the user wearing the AR device onto the real-scene image; A display module is used to display the first augmented reality image and the at least one frame of the second augmented reality image through the AR device.

12. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and computer instructions stored in the memory and executable by the processor, and the processor implements the method according to any one of claims 1 to 10 when executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images