Three-dimensional scanning method and apparatus, device, and storage medium

Abstract

A three-dimensional scanning method and apparatus, a device, and a storage medium. The method comprises: acquiring first scanned image data corresponding to a first area of a target object; generating a first model on the basis of the first scanned image data, wherein at least partial area of the first model is locked; in a continuous scanning process of the target object, at least acquiring second scanned image data corresponding to a second area different from the locked area in the first model; generating a target model on the basis of the second scanned image data and the first model; and optimizing the second scanned image data in real time on the basis of the first scanned image data corresponding to the locked area in the first model, and updating the target model.

Description

3D scanning methods, apparatus, equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410495674.7, filed on April 24, 2024, entitled “Three-dimensional scanning method, apparatus, device and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of computer technology, and specifically relates to a three-dimensional scanning method, apparatus, device and storage medium. Background Technology

[0003] 3D scanning can scan the spatial shape, structure, and color of an object to obtain a corresponding 3D model, and it is widely used in medical, educational, and industrial fields.

[0004] With the increasing demand for 3D scanning, secondary scanning may be required in various scenarios such as oral cavity scanning, facial scanning, person scanning, prosthetic limb scanning, artifact scanning, industrial scanning, and architectural scanning. However, misalignment may occur when stitching the first and second scanned image data together. Therefore, there is an urgent need for a 3D scanning method that can reduce the risk of misalignment. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a three-dimensional scanning method, apparatus, device, and storage medium.

[0006] A first aspect of this disclosure provides a three-dimensional scanning method, the method comprising: acquiring first scan image data corresponding to a first region of a target object; generating a first model based on the first scan image data, wherein at least a portion of the first model is locked; during continued scanning of the target object, acquiring second scan image data corresponding to at least a second region of the target object that is different from the locked region in the first model; generating a target model based on the second scan image data and the first model; and optimizing the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model, and updating the target model.

[0007] A second aspect of this disclosure provides a three-dimensional scanning apparatus, comprising: a first acquisition module configured to acquire first scan image data corresponding to a first region of a target object; a first generation module configured to generate a first model based on the first scan image data, wherein at least a portion of the first model is locked; a second acquisition module configured to, during continued scanning of the target object, acquire at least second scan image data corresponding to a second region of the target object that is different from the locked region in the first model; a second generation module configured to generate a target model based on the second scan image data and the first model; and a first optimization module configured to optimize the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model, and update the target model.

[0008] A third aspect of this disclosure provides an electronic device, the server comprising: a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of the first aspect described above.

[0009] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the method of the first aspect described above.

[0010] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0011] This embodiment of the disclosure is capable of acquiring first scanned image data corresponding to a first region of a target object; generating a first model based on the first scanned image data, wherein at least a portion of the first model is locked; during continued scanning of the target object, acquiring second scanned image data corresponding to at least a second region of the target object that is different from the locked region in the first model; generating a target model based on the second scanned image data and the first model; and optimizing the second scanned image data in real time based on the first scanned image data corresponding to the locked region in the first model, and updating the target model. It is evident that by employing the above technical solution, the second scanned image data can be optimized to improve its quality, thereby reducing the risk of stitching misalignment and improving the 3D scanning effect. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0014] Figure 1 is a flowchart of a three-dimensional scanning method provided in an embodiment of this disclosure;

[0015] Figure 2 is a schematic diagram of a complete three-dimensional model corresponding to a first region provided in an embodiment of this disclosure;

[0016] Figure 3 is a schematic diagram of a three-dimensional model corresponding to a first region and a new region provided in an embodiment of this disclosure;

[0017] Figure 4 is a flowchart of another three-dimensional scanning method provided in an embodiment of this disclosure;

[0018] Figure 5 is a schematic diagram of the structure of a three-dimensional scanning device provided in an embodiment of this disclosure;

[0019] Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0022] Figure 1 is a flowchart of a three-dimensional scanning method provided in this embodiment of the present disclosure. This method can be executed by an electronic device. The method is applicable to three-dimensional scanners such as dental scanners, facial scanners, industrial scanners, professional scanners, handheld scanners, and fixed scanners, and can realize three-dimensional reconstruction of objects or scenes such as teeth, faces, bodies, industrial products, industrial equipment, cultural relics, artworks, prostheses, medical instruments, and buildings. The electronic device can be exemplarily understood as a device such as a 3D scanner, mobile phone, tablet computer, laptop computer, desktop computer, or smart TV. As shown in Figure 1, the method provided in this embodiment includes the following steps:

[0023] S110. Obtain the first scan image data corresponding to the first region of the target object.

[0024] In this embodiment of the disclosure, a user can use a 3D scanner to scan a first region of the target object to obtain first scanned image data, thereby enabling the electronic device to acquire the first scanned image data. It should be noted that the first region can be any region on the target object, and this disclosure does not limit it.

[0025] S120. Generate a first model based on the first scanned image data, wherein at least a portion of the first model is locked.

[0026] In this embodiment of the disclosure, after the electronic device acquires the first scanned image data, it can generate a first model based on the first scanned image data.

[0027] Specifically, the first model is a three-dimensional model. The first model can be generated based on the first scanned image data in any known way, such as by fusion (e.g., tracking fusion of previous and next frames), AI generation, etc., but is not limited to these methods.

[0028] For example, Figure 2 is a schematic diagram of a complete three-dimensional model corresponding to a first region provided in an embodiment of this disclosure. Referring to Figure 2, in a planting pole scanning scenario, a three-dimensional scanner is used to scan the first region (the region other than where the planting pole is located). The electronic device fuses the acquired first scan image data to obtain a complete three-dimensional model corresponding to the first region.

[0029] Specifically, at least a portion of the region in the first model is locked, meaning the locked region remains unchanged, and the first scan image data corresponding to the locked region (i.e., the first scan image data used to reconstruct the locked region) remains unchanged. Therefore, the first scan image data corresponding to the locked region is the locked data.

[0030] Specifically, what data is included in "the first scanned image data corresponding to the locked area"? Typical examples will be given below, but this does not constitute a limitation of this disclosure.

[0031] In some embodiments, the first scanned image data corresponding to the locked region in the first model includes: all data in the first scanned image data; or: data in the first scanned image data that the user has selected to lock; or: data corresponding to the region of interest in the first scanned image data.

[0032] Specifically, users can select locked data from the first scanned image data in any possible way. For example, users can select at least a portion of the data from the first scanned image data using a mouse, keyboard, or touch panel. In this way, the electronic device can identify the selected data as locked data, but it is not limited to this.

[0033] Specifically, the region of interest is a region within the target object that contains a target object, a feature, or any part that needs further study, measurement, identification, or manipulation. The region of interest may be recorded in the scanning order, but is not limited to this.

[0034] Specifically, the region of interest can be determined in any possible way. For example, for the first scanned image data, AI recognition, feature extraction and matching, or threshold segmentation can be used to determine the data corresponding to the region of interest.

[0035] For example, in a dental scanning scenario, if the scanning order specifies the region of interest as tooth number 1, the electronic device can use AI to identify the data corresponding to tooth number 1 in the first scanned image data and lock that data. Similarly, in an industrial scanning scenario, if the scanning order specifies the region of interest as industrial equipment A, the electronic device can use AI to identify the data corresponding to industrial equipment A in the first scanned image data and lock that data. Likewise, in a large-scale building scene scanning scenario, if the scanning order specifies the region of interest as building number 1, the electronic device can use AI to identify the data corresponding to building number 1 in the first scanned image data and lock that data.

[0036] Understandably, locking all data in the first scanned image data simplifies and speeds up the process of determining the locked data. Locking the user-selected data in the first scanned image data allows for flexible selection of locked data, making the locked data more aligned with user needs. Locking the data corresponding to the region of interest in the first scanned image data locks the data corresponding to areas of greater interest, facilitating further research on these areas by the user.

[0037] S130. During the continued scanning of the target object, at least the second scan image data corresponding to the second region of the target object that is different from the locked region in the first model is obtained.

[0038] In this embodiment of the disclosure, after scanning the first area of ​​the target object with a 3D scanner, the user can continue to scan the second area. During the scanning of the second area, second scan image data is obtained, so that the electronic device can acquire the second scan image data.

[0039] Optionally, the second region may include: a new region in the target object that is different from the first region; and / or: an area in the first region that is not locked.

[0040] Specifically, the new region and the first region do not overlap; they are two different regions within the target object.

[0041] Specifically, the unlocked area in the first region refers to the remaining area in the first region other than the area corresponding to the locked area on the target object.

[0042] Optionally, S130 may include: obtaining the current scan frame in real time during the continued scanning of the target object; if the current scan frame corresponds entirely to the locked area in the first model, then determining that the current scan frame will not participate in updating the first model; if some data in the current scan frame corresponds to the locked area in the first model, then determining that the data in the current scan frame corresponding to the locked area in the first model will not participate in updating the first model, and determining that the remaining data in the current scan frame other than the data corresponding to the locked area in the first model is the second scan image data; if no data in the current scan frame corresponds to the locked area in the first model, then determining that the current scan frame is the second scan image data.

[0043] Specifically, if the region corresponding to the current scan frame on the target object is the same region corresponding to the locked region on the target object, then the current scan frame corresponds entirely to the locked region in the first model. In this case, the current scan frame does not participate in updating the first model; it can be deleted or retained.

[0044] Specifically, if the area corresponding to the target object in the current scan frame overlaps with the area corresponding to the locked area in the target object, and there are also non-overlapping areas, then the data corresponding to the overlapping area in the current scan frame corresponds to the locked area in the first model and does not participate in updating the first model. The data corresponding to the non-overlapping area in the current scan frame (i.e., the remaining data) does not correspond to the locked area in the first model and belongs to the second scan image data.

[0045] Specifically, if the region corresponding to the current scan frame on the target object does not overlap with the region corresponding to the locked region on the target object, then the current scan frame does not correspond to the locked region in the first model. In this case, all data in the current scan frame belongs to the second scan image data.

[0046] It is understandable that during the continued scanning of the target object, the scanning frame corresponding to the first region may be obtained again. In this embodiment of the present disclosure, by setting a judgment on whether the current scanning frame corresponds to the locked region, and determining the data that does not correspond to the locked region as the second scanning image data, the second scanning image data can be made to exclude the data corresponding to the locked region, which is beneficial to improving the accuracy of the second scanning image data.

[0047] S140. Generate a target model based on the second scanned image data and the first model.

[0048] In this embodiment of the disclosure, while the locked data in the first scanned image data remains unchanged, the electronic device can stitch together the locked data in the first scanned image data and the second scanned image data. In other words, the area in the first model other than the locked area is updated based on the second scanned image data, thereby generating the target model.

[0049] In some embodiments, the second region includes only a new region in the target object that is different from the first region. In this case, S140 includes: S141, while keeping the locked region in the first model unchanged, updating the first model based on the second scan image data corresponding to the new region in the target object that is different from the first region, thereby generating the target model.

[0050] In some other embodiments, the second region includes only the unlocked region in the first region. In this case, S140 includes: S142, updating the first model based on the second scan image data corresponding to the unlocked region in the first region, thereby generating the target model.

[0051] In some other embodiments, the second region includes a new region in the target object that is different from the first region and an unlocked region in the first region. In this case, S140 includes: S143, while keeping the locked region in the first model unchanged, updating the first model based on the second scan image data corresponding to the new region in the target object that is different from the first region, and updating the first model based on the second scan image data corresponding to the unlocked region in the first region, thereby generating the target model.

[0052] S150. Based on the first scan image data corresponding to the locked area in the first model, the second scan image data is optimized in real time, and the target model is updated.

[0053] In this embodiment of the disclosure, while the locked data in the first scanned image data remains unchanged, the electronic device can optimize the second scanned image data in real time based on the locked data in the first scanned image data. Furthermore, the electronic device can stitch together the locked data in the first scanned image data and the real-time optimized second scanned image data to update the target model.

[0054] Specifically, the second scanned image data can be optimized in any possible way, such as by using AI optimization, but it is not limited to this.

[0055] Specifically, the locked data in the first scanned image data and the real-time optimized second scanned image data can be stitched together in any known way. For example, fusion (such as frame-to-frame tracking fusion) or AI stitching can be used to stitch the locked data in the first scanned image data and the real-time optimized second scanned image data together, but it is not limited to these methods.

[0056] Understandably, without optimization of the second scanned image data, cumulative errors will occur as the target object is continuously scanned. These cumulative errors are built up from the errors of the previously scanned image data; for example, the tenth frame of data will accumulate the errors from the previous nine frames. Therefore, the cumulative error will increase over time. If the cumulative error is large, stitching the newly arrived scanned image data with the already scanned image data may fail, affecting scanning smoothness. In this embodiment, real-time optimization of the second scanned image data is added. This reduces or eliminates cumulative errors, lowers the risk of stitching failure when stitching the newly arrived scanned image data with the already scanned image data, and improves scanning smoothness.

[0057] For example, Figure 3 is a schematic diagram of a three-dimensional model corresponding to a first region and a new region provided in an embodiment of this disclosure. Referring to Figure 3, after the first region is scanned, the three-dimensional scanning model is used to scan the region where the planting pole is located (i.e., the new region). The electronic device receives the second scan image data corresponding to the new region sent by the three-dimensional scanner, and optimizes the second scan image data based on the data locked in the first scan image data. The target model is updated based on the real-time optimized second scan image data, so that the target model includes the region corresponding to the region where the planting pole is located. As can be seen from Figure 3, the stitching effect between the first region and the new region is good, and there is no stitching misalignment problem.

[0058] In this embodiment, the locked area in the first scanned image data can be kept unchanged while the second scanned image data is optimized. This improves the quality of the scanned image data corresponding to the new area (i.e., the second scanned image data), thereby reducing the risk of misalignment between the first and new areas and improving the 3D scanning effect.

[0059] In some embodiments, before locking the locked region in the first model, the method further includes: deleting the first scan image data corresponding to at least a portion of the region in the first model.

[0060] It should be noted that the deletion operation of the first scan image data corresponding to at least a portion of the region from the first model in the embodiments of this application has multiple implementation schemes, including but not limited to: one implementation scheme is to separate the first scan image data corresponding to at least a portion of the region from the first model, and only retain the first model after separating the first scan image data corresponding to at least a portion of the region in the display interface. In this implementation scheme, the deletion operation is a selective retention of the display result, rather than completely deleting the first scan image data corresponding to at least a portion of the region, that is, only deleting the first scan image data corresponding to at least a portion of the region from the display interface; another implementation scheme is to directly and completely delete the first scan image data corresponding to at least a portion of the region from the source data; yet another implementation scheme is to separate the first scan image data corresponding to at least a portion of the region from the first model and increase the transparency of the separated first scan image data corresponding to at least a portion of the region, such as displaying it with semi-transparency, so as to achieve the effect of highlighting the remaining first model in the display interface; yet another implementation scheme is to delete the first scan image data corresponding to at least a portion of the region from the intermediate product of the reconstructed model or the reconstructed model, so that the display interface does not display the first scan image data corresponding to at least a portion of the region.

[0061] Optionally, the first scanned image data corresponding to the deleted region in the first model includes: data in the first scanned image data that the user selects to delete; and / or: data corresponding to the preset ignored region of the target object in the first scanned image data.

[0062] Specifically, users can select the data to be deleted from the first scanned image data in any possible way. For example, users can select at least a portion of the data from the first scanned image data using a mouse, keyboard, or touch panel. In this way, the electronic device can identify the selected data as the data to be deleted, but it is not limited to this.

[0063] Specifically, the data corresponding to the preset ignored region can be determined from the first scanned image data in any possible way. For example, for the first scanned image data, AI recognition, feature extraction and matching, or threshold segmentation can be used to determine the data corresponding to the preset ignored region.

[0064] Specifically, after deleting the first scan image data corresponding to at least a portion of the regions in the first model, the first scan image data corresponding to the locked regions in the first model includes: all data in the remaining first scan image data; or; data in the remaining first scan image data that the user selected to delete; or; data corresponding to the region of interest in the remaining first scan image data.

[0065] It is understandable that by deleting the first scan image data corresponding to at least a portion of the area in the first model before setting the locked area, some of the first scan image data can be deleted, making it easier to set the locked area later.

[0066] In some embodiments, after locking the locked region in the first model and before acquiring the second scan image, the first scan image data corresponding to at least a portion of the other regions in the first model besides the locked region is deleted.

[0067] For a detailed understanding of "the first scanned image data corresponding to the deleted region in the first model", please refer to the previous text, which will not be repeated here.

[0068] It is understandable that by deleting the first scan image data corresponding to at least a portion of the regions in the first model other than the locked region after setting the locked region, useless first scan image data can be deleted to reduce the occupation of storage resources.

[0069] In some embodiments, the method further includes: saving a first model prior to locking the locked area; and / or;

[0070] Save the first model before deleting the area to be deleted; and / or save the first model before acquiring the second scan image data.

[0071] Understandably, saving the first model at each stage facilitates subsequent tracing and analysis of the first model at each stage. The saving process can be performed in real time or not.

[0072] In some embodiments, the method further includes: displaying a first model generated based on first scanned image data; and / or: displaying an updated first model in real time after the first model is updated; and / or: displaying a target model generated based on the first model and second scanned image data; and / or: displaying an updated target model in real time after the target model is updated.

[0073] Understandably, by fusing the first scanned image data in real time to obtain the first model and displaying it, users can intuitively, vividly, and in real time see the scanning effect of the first region. Furthermore, when the first model is updated due to reasons such as the deletion of deleted areas, displaying the updated first model in real time allows users to intuitively, vividly, and in real time see the changes in the first model.

[0074] Understandably, by generating and displaying the target model in real time based on the first and second scanned image data, users can intuitively, visually, and in real-time observe the scanning effect of the second region and easily understand the current scanning progress. Furthermore, when the target model is updated due to optimization of the second scanned image data, displaying the updated target model in real time allows users to intuitively, visually, and in real-time observe the changes in the target model.

[0075] Figure 4 is a flowchart illustrating another three-dimensional scanning method provided in this embodiment. This embodiment optimizes the above embodiments and can be combined with various optional solutions from one or more of the above embodiments.

[0076] As shown in Figure 4, the three-dimensional scanning method may include the following steps.

[0077] S410. Obtain the first scan image data corresponding to the first region of the target object.

[0078] In this embodiment of the disclosure, during the scanning of the first region, the camera in the 3D scanner can acquire multiple first scan frames of the first region of the target object. The pose measurement module (e.g., inertial measurement unit measurement) in the 3D scanner can measure the pose corresponding to each first scan frame (referred to as the first measurement pose). Thus, the electronic device can receive the multiple first scan frames sent by the 3D scanner, as well as the first measurement poses corresponding to the multiple first scan frames. For example, the pose can represent the pose transformation matrix (RT) of the current scan frame from the local coordinate system (camera coordinate system) to the world coordinate system (global coordinate system).

[0079] Specifically, the first pose can be the first measured pose, or it can be the pose obtained by optimizing the first measured pose. This application does not limit this.

[0080] Furthermore, similar content in S410 and S110 will not be repeated here.

[0081] S420. Generate a first model based on the first scanned image data, wherein at least a portion of the first model is locked.

[0082] Specifically, the first model is obtained by fusing the acquired first scan frame and its corresponding first pose.

[0083] Furthermore, similar content in S420 and S120 will not be repeated here.

[0084] S430. During the continued scanning of the target object, at least the second scan image data corresponding to the second region of the target object that is different from the locked region in the first model is obtained.

[0085] In this embodiment of the disclosure, after the first region of the target object is scanned, the second region can continue to be scanned. During the scanning of the second region, the camera in the 3D scanner can acquire at least one second scan frame by capturing images of the second region of the target object. The pose measurement module in the 3D scanner can obtain the pose (i.e., the second pose) corresponding to each second scan frame. Thus, the electronic device can receive at least one second scan frame sent by the camera and the second pose corresponding to each second scan frame.

[0086] Furthermore, similar content to S430 and S130 will not be repeated here.

[0087] S440. Generate a target model based on the second scanned image data and the first model.

[0088] Specifically, S440 is similar to S140, and will not be described in detail here.

[0089] S450. Based on the first scan image data and the second scan image data, determine the inter-frame error of the second scan frame.

[0090] Specifically, the inter-frame error of a second scan frame is used to characterize the overall distance between feature point pairs in the second scan frame and its corresponding associated scan frame, wherein there is an overlapping image region between the associated scan frame and the second scan frame, and the associated scan frame of a second scan frame may include the first scan frame and / or the second scan frame.

[0091] In some embodiments, S450 may include: inputting the second scan frame and its corresponding associated scan frame into a pre-trained inter-frame error detection model to obtain the inter-frame error of the second scan frame.

[0092] In some other embodiments, S450 may include: S451, for the second scan frame and each associated scan frame therewith, determining the inter-frame error between the two based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame.

[0093] Specifically, the inter-frame error of a second scan frame is used to characterize the distance between feature point pairs in the second scan frame and its corresponding associated scan frame.

[0094] In one example, the associated scan frame includes a first associated scan frame. Accordingly, S451 may include: S4511, for the second scan frame and each corresponding first associated scan frame, determining the inter-frame error between the two based on the second pose corresponding to the second scan frame and the second pose corresponding to the first associated scan frame.

[0095] Specifically, the first associated scan frame is another second scan frame that has an image region overlapping with the second scan frame. The specific number of the first associated scan frames can be set by those skilled in the art according to the actual situation, and is not limited here. For example, the first associated scan frames corresponding to one second scan frame are: the N1 second scan frames adjacent to it before it, and / or the N2 second scan frames adjacent to it after it, where N1 and N2 are both positive integers.

[0096] In another example, the associated scan frame includes a second associated scan frame. Accordingly, S451 may include: S4512, for the second scan frame and each corresponding second associated scan frame, determining the inter-frame error between the two based on the second pose corresponding to the second scan frame and the second pose corresponding to the second associated scan frame.

[0097] Specifically, the second associated scan frame is the first scan frame that has an image region overlapping with the second scan frame. The specific number of the second associated scan frames can be set by those skilled in the art according to the actual situation, and is not limited here.

[0098] In yet another example, the associated scan frame includes a first associated scan frame and a second associated scan frame, and accordingly, S451 may include S4511 and S4512.

[0099] Optionally, based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame, the inter-frame error between the two is determined, including: based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame, the distance between the feature point pairs in the two is determined, and the inter-frame error is determined based on the distance.

[0100] Further optionally, the feature point pairs include marker feature point pairs, texture feature point pairs, and / or geometric feature point pairs. Accordingly, determining the distance between the feature point pairs based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame may include: calculating a first distance between the marker feature point pairs, a second distance between the texture feature point pairs, and / or a third distance between the geometric feature point pairs based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame; if the feature point pair only includes marker feature point pairs (texture feature point pairs, or geometric feature point pairs), then the first distance (second distance, or third distance) is used as the inter-frame error; if the feature point pair includes both marker feature point pairs and texture feature point pairs, then... The inter-frame error is obtained by directly summing the first and second distances or by weighted summation. If the feature point pair includes both marker feature point pairs and geometric feature point pairs, the inter-frame error is obtained by directly summing the first and third distances or by weighted summation. If the feature point pair includes both texture feature point pairs and geometric feature point pairs, the inter-frame error is obtained by directly summing the second and third distances or by weighted summation. If the feature point pair includes both marker feature point pairs, texture feature point pairs, and geometric feature point pairs, the inter-frame error is obtained by directly summing the first, second, and third distances or by weighted summation.

[0101] Specifically, the marker feature point pair consists of marker points pre-pasted on the target object.

[0102] Specifically, a texture feature point pair is composed of texture feature points on the target object; for example, a texture feature can be a color feature.

[0103] Specifically, the geometric feature point pair is composed of geometric feature points on the target object. For example, the geometric feature can be the point cloud feature of the target object.

[0104] Specifically, the first distance, the second distance, and the third distance can be Euclidean distance, the distance from a point to a tangent plane, or a feature distance.

[0105] Understandably, using at least one feature point pair to calculate the distance and fusing the distances of various feature point pairs can ensure the accuracy of the inter-frame error calculation while improving the flexibility of the calculation method.

[0106] S452. Determine the inter-frame error of the second scan frame based on the inter-subframe error between the second scan frame and its corresponding associated scan frames.

[0107] In one example, the associated scan frame includes a first associated scan frame. Accordingly, S452 may include: directly summing or weighting the inter-frame errors between the second scan frame and each of the corresponding first associated scan frames to obtain the inter-frame error of the second scan frame.

[0108] In another example, the associated scan frame includes a second associated scan frame. Accordingly, S452 may include: directly summing or weighting the inter-frame errors between the second scan frame and each of its corresponding second associated scan frames to obtain the inter-frame error of the second scan frame.

[0109] In another example, the associated scan frames include a first associated scan frame and a second associated scan frame. Accordingly, S452 may include: directly summing or weighting the inter-frame errors between the second scan frame and each of its corresponding first associated scan frames to obtain a first summed value; directly summing or weighting the inter-frame errors between the second scan frame and each of its corresponding second associated scan frames to obtain a second summed value; and directly summing or weighting the first summed value and the second summed value, and then averaging them to obtain the inter-frame error of the second scan frame.

[0110] In another example, the associated scan frames include a first associated scan frame and a second associated scan frame. Accordingly, S452 may include: directly averaging, summing, or weighting the inter-frame errors between the second scan frame and each of its corresponding first and second associated scan frames to obtain the inter-frame error of the second scan frame.

[0111] It can be understood that when calculating the inter-frame error of the second scan frame, the inter-subframe error between the second scan frame and each associated scan frame is determined, and the inter-frame error between the second scan frame and each associated scan frame is fused to obtain the inter-frame error of the second scan frame. This calculation method can ensure the accuracy of the inter-frame error calculation while reducing the calculation difficulty.

[0112] S460. Determine the global error based on the inter-frame error of each second scan frame.

[0113] Specifically, the global error is the average or sum of the inter-frame errors corresponding to each second scan frame.

[0114] Specifically, the inter-frame errors of each second scan frame are averaged, directly summed, or weighted and summed to obtain the global error.

[0115] S470. If the global error is greater than the preset error threshold, the second scan image data is optimized in real time based on the first scan image data corresponding to the locked area in the first model, and the target model is updated.

[0116] Specifically, those skilled in the art can set specific values ​​for the preset error threshold according to actual circumstances, and this disclosure does not limit this.

[0117] In some examples, real-time optimization of the second scan image data based on the first scan image data corresponding to the locked region in the first model may include: adjusting the second pose until the global error is less than or equal to a preset error threshold.

[0118] Specifically, the second pose is calculated and adjusted. Based on the first scan image data and the second scan image data after adjusting the second pose, the global error is re-determined. It is then determined whether the global error corresponding to the optimization is less than or equal to a preset error threshold. If not, the calculation and adjustment of the second pose continues until the global error corresponding to the adjusted second pose is less than or equal to the preset error threshold, thus obtaining the optimized second scan image data. The calculation and adjustment of the second pose includes recalculating the inter-frame relative coordinate transformation matrix (inter-frame relative RT) between each second scan frame and its corresponding associated scan frames in the second scan image data based on the global error, to adjust the coordinate transformation matrix (overall RT) of each second scan frame.

[0119] In other examples, the first scan image data and the second scan image data are input into a pre-trained optimization model so that the optimization model can optimize the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model.

[0120] This embodiment of the disclosure can determine the global error based on the first scanned image data and the second scanned image data. If the global error is greater than a preset error threshold, the second pose is optimized based on the locked data in the first scanned image data and the second scanned image data. This allows for timely optimization of the second pose, preventing the global error from accumulating and increasing. Thus, the optimized second scanned image data can be re-fused in a timely manner to obtain an optimized 3D model corresponding to the new region, providing users with high-quality scanning results. The optimized 3D model corresponding to the new region can be displayed independently or stitched together with the locked region in the first model to obtain and display a completely new 3D model.

[0121] Figure 5 is a schematic diagram of a three-dimensional scanning device provided in an embodiment of this disclosure. This three-dimensional scanning device can be understood as the aforementioned electronic device or a functional module within the aforementioned electronic device. As shown in Figure 5, the three-dimensional scanning device 500 includes: a first acquisition module 510 configured to acquire first scan image data corresponding to a first region of a target object; a first generation module 520 configured to generate a first model based on the first scan image data, wherein at least a portion of the first model is locked; a second acquisition module 530 configured to, during continued scanning of the target object, acquire at least second scan image data corresponding to a second region of the target object that is different from the locked region in the first model; a second generation module 540 configured to generate a target model based on the second scan image data and the first model; and a first optimization module 550 configured to optimize the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model, and update the target model.

[0122] In some embodiments, the first scanned image data corresponding to the locked region in the first model includes: all data in the first scanned image data; or: data in the first scanned image data that the user has selected to lock; or: data corresponding to the region of interest in the first scanned image data.

[0123] In some embodiments, the second region includes: a new region in the target object that is different from the first region; and / or: an unlocked region in the first region.

[0124] In some embodiments, the second acquisition module 530 is specifically configured to: obtain the current scan frame in real time during the continued scanning of the target object; if the entire current scan frame corresponds to the locked area in the first model, then determine that the current scan frame will not participate in updating the first model; if some data in the current scan frame corresponds to the locked area in the first model, then determine that the data in the current scan frame corresponding to the locked area in the first model will not participate in updating the first model, and determine that the remaining data in the current scan frame other than the data corresponding to the locked area in the first model is the second scan image data; if no data in the current scan frame corresponds to the locked area in the first model, then determine that the current scan frame is the second scan image data.

[0125] In some embodiments, the second generation module 540 is specifically configured to update the first model based on the second scan image data corresponding to the unlocked area in the first region while keeping the locked area in the first model unchanged, and / or update the first model based on the second scan image data corresponding to the new region in the target object that is different from the first region, thereby generating a target model.

[0126] In some embodiments, the apparatus further includes a deletion module, specifically configured to: delete first scan image data corresponding to at least a portion of the regions in the first model before locking the locked regions in the first model; or; delete first scan image data corresponding to at least a portion of the regions in other regions of the first model besides the locked regions after locking the locked regions in the first model and before acquiring the second scan image.

[0127] In some embodiments, the first scanned image data corresponding to the deleted region in the first model includes: data in the first scanned image data that the user selects to delete; and / or: data corresponding to a preset ignored region of the target object in the first scanned image data.

[0128] In some embodiments, the device further includes a storage module, specifically configured to: store a first model before locking the locked area; and / or; store a first model before deleting the deleted area; and / or; store a first model before acquiring the second scan image data.

[0129] In some embodiments, the device further includes a display module, specifically configured to: display a first model generated based on first scanned image data; and / or: display an updated first model in real time after the first model is updated; and / or: display a target model generated based on second scanned image data and the first model; and / or: display an updated target model in real time after the target model is updated.

[0130] In some embodiments, the first scan image data includes multiple first scan frames and a first pose corresponding to the multiple first scan frames, and the second scan image data includes at least one second scan frame and a second pose corresponding to at least one second scan frame; wherein, the first optimization module 550 includes: a first determination submodule configured to determine the inter-frame error of the second scan frame based on the first scan image data and the second scan image data; a second determination submodule configured to determine the global error based on the inter-frame error of each second scan frame; and a first optimization submodule configured to perform real-time optimization of the second scan image data based on the first scan image data corresponding to the locked region in the first model if the global error is greater than a preset error threshold.

[0131] In some embodiments, the first determining submodule includes: a first determining unit configured to determine, for a second scan frame and each associated scan frame therewith, an inter-frame error between the second scan frame and the associated scan frame based on a second pose corresponding to the second scan frame and a pose corresponding to the associated scan frame, wherein the associated scan frame includes: a first associated scan frame and / or a second associated scan frame, the first associated scan frame being another second scan frame having an image region overlapping with the second scan frame, and the second associated scan frame being a first scan frame having an image region overlapping with the second scan frame; and a second determining unit configured to determine the inter-frame error of the second scan frame based on the inter-frame error between the second scan frame and each associated scan frame therewith.

[0132] In some embodiments, the first determining unit is specifically configured to, for the second scan frame and each associated scan frame therewith, determine the distance between feature point pairs in the two based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame, and determine the inter-frame error based on the distance.

[0133] In some embodiments, feature point pairs include marker feature point pairs, texture feature point pairs, and / or geometric feature point pairs.

[0134] The first optimization submodule is specifically configured to adjust the second pose if the global error is greater than a preset error threshold, until the global error is less than or equal to the preset error threshold.

[0135] The apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0136] This disclosure also provides an electronic device, which includes: a memory storing a computer program; and a processor configured to execute the computer program, wherein when the computer program is executed by the processor, it can implement the methods of any of the above embodiments.

[0137] For example, Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Referring specifically to Figure 6 below, it shows a schematic diagram of the structure suitable for implementing the electronic device 600 in the embodiments of this disclosure. The electronic device 600 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 6 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0138] As shown in Figure 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0139] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 illustrates electronic device 600 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0140] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0141] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0142] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0143] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0144] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire first scanned image data corresponding to a first region of a target object; generate a first model based on the first scanned image data, wherein at least some regions in the first model are locked; during continued scanning of the target object, acquire second scanned image data corresponding to at least a second region of the target object that is different from the locked region in the first model; generate a target model based on the second scanned image data and the first model; and optimize the second scanned image data in real time based on the first scanned image data corresponding to the locked region in the first model, and update the target model.

[0145] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0148] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0150] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Industrial applicability

[0153] The 3D scanner disclosed herein can acquire first scanned image data corresponding to a first region of a target object; generate a first model based on the first scanned image data, wherein at least a portion of the first model is locked; during continued scanning of the target object, acquire second scanned image data corresponding to at least a second region of the target object that is different from the locked region in the first model; generate a target model based on the second scanned image data and the first model; optimize the second scanned image data in real time based on the first scanned image data corresponding to the locked region in the first model, and update the target model. It is evident that by adopting the above technical solution, the second scanned image data can be optimized to improve its quality, thereby reducing the risk of stitching misalignment, improving the 3D scanning effect, and possessing strong industrial applicability.

Claims

1. A three-dimensional scanning method, wherein, include: Obtain the first scanned image data corresponding to the first region of the target object; A first model is generated based on the first scanned image data, wherein at least a portion of the first model is locked. During the continued scanning of the target object, at least the second scan image data corresponding to the second region of the target object that is different from the locked region in the first model is obtained; A target model is generated based on the second scanned image data and the first model; The second scan image data is optimized in real time based on the first scan image data corresponding to the locked area in the first model, and the target model is updated.

2. The method according to claim 1, wherein, The first scan image data corresponding to the locked region in the first model includes: All data in the first scanned image data; or; The data in the first scanned image data that the user has selected to lock; or; The data corresponding to the region of interest in the first scanned image data.

3. The method according to claim 1, wherein, The second region includes: A new region in the target object that is different from the first region; and / or; The first region is an unlocked region.

4. The method according to claim 3, wherein, During the continued scanning of the target object, acquiring at least the second scan image data corresponding to regions of the target object that are different from the first region includes: During the continued scanning of the target object, the current scan frame is obtained in real time; If all of the current scan frames correspond to the locked regions in the first model, then it is determined that the current scan frames will not participate in updating the first model. If some data in the current scan frame corresponds to a locked area in the first model, then it is determined that the data in the current scan frame corresponding to the locked area in the first model will not participate in updating the first model, and the remaining data in the current scan frame other than the data corresponding to the locked area in the first model will be the second scan image data. If the current scan frame does not contain data corresponding to the locked area in the first model, then the current scan frame is determined to be the second scan image data.

5. The method according to claim 3, wherein, Generate a target model based on the second scanned image data and the first model, including: While keeping the locked region in the first model unchanged, the first model is updated based on the second scan image data corresponding to the unlocked region in the first region, and / or the first model is updated based on the second scan image data corresponding to a new region in the target object that is different from the first region, thereby generating the target model.

6. The method according to claim 1, wherein, Before locking the locked region in the first model, the method further includes: deleting the first scan image data corresponding to at least a portion of the region in the first model; or; After locking the locked region in the first model and before acquiring the second scan image, delete the first scan image data corresponding to at least a portion of the other regions in the first model besides the locked region.

7. The method according to claim 6, wherein, The first scan image data corresponding to the deleted region in the first model includes: Data in the first scanned image data that the user selected to delete; and / or; The data corresponding to the preset ignore region of the target object in the first scanned image data.

8. The method according to claim 6, wherein, Also includes: Save the first model before locking the locked area; and / or; Save the first model before deleting the region; and / or; Save the first model before acquiring the second scanned image data.

9. The method according to claim 6, wherein, Also includes: Display the first model generated based on the first scanned image data; and / or; After the first model is updated, the updated first model is displayed in real time; and / or; Display the target model generated based on the second scanned image data and the first model; and / or; After the target model is updated, the updated target model is displayed in real time.

10. The method according to any one of claims 1 to 9, wherein, The first scan image data includes multiple first scan frames and a first pose corresponding to the multiple first scan frames; the second scan image data includes at least one second scan frame and a second pose corresponding to the at least one second scan frame. The step of optimizing the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model includes: Based on the first scanned image data and the second scanned image data, the inter-frame error of the second scanned frame is determined; The global error is determined based on the inter-frame error of each of the second scan frames; If the global error is greater than a preset error threshold, the second scan image data is optimized in real time based on the first scan image data corresponding to the locked region in the first model.

11. The method according to claim 10, wherein, The step of determining the inter-frame error of the second scan frame based on the first scan image data and the second scan image data includes: For the second scan frame and each associated scan frame therewith, based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame, the inter-frame error between them is determined. The associated scan frame includes: a first associated scan frame and / or a second associated scan frame. The first associated scan frame is another second scan frame that has an image region overlapping with the second scan frame, and the second associated scan frame is a first scan frame that has an image region overlapping with the second scan frame. The inter-frame error of the second scan frame is determined based on the inter-subframe error between the second scan frame and each of the associated scan frames therecorresponding to it.

12. The method according to claim 11, wherein, The step of determining the inter-subframe error between the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame includes: Based on the second pose corresponding to the second scan frame and the pose corresponding to the associated scan frame, the distance between the feature point pairs in the two is determined, and the inter-frame error is determined based on the distance.

13. The method according to claim 12, wherein, The feature point pairs include marker feature point pairs, texture feature point pairs, and / or geometric feature point pairs.

14. The method of claim 10, wherein, The real-time optimization of the second scan image data based on the first scan image data corresponding to the locked region in the first model includes: Adjust the second pose until the global error is less than or equal to the preset error threshold.

15. A three-dimensional scanning device, wherein, include: The first acquisition module is configured to acquire the first scan image data corresponding to the first region of the target object; The first generation module is configured to generate a first model based on the first scanned image data, wherein at least a portion of the first model is locked. The second acquisition module is configured to acquire, during the continued scanning of the target object, at least the second scan image data corresponding to a second region in the target object that is different from the locked region in the first model. The second generation module is configured to generate a target model based on the second scanned image data and the first model. The first optimization module is configured to optimize the second scan image data in real time based on the first scan image data corresponding to the locked region in the first model, and update the target model.

16. An electronic device, wherein, include: A processor and a memory, wherein the memory stores a computer program that, when executed by the processor, performs the method of any one of claims 1-14.

17. A computer-readable storage medium, wherein, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-14.

Images